crtc - a generic-purposed expect script to automate interactive tasks

Table of Content

• 1. quick examples
  • 1.1. automate device login process
    • 1.1.1. configure login info for individual device
    • 1.1.2. configure login info for a group of device
  • 1.2. host name resolution
  • 1.3. automate commands after login
    • 1.3.1. interupting an automation loop
    • 1.3.2. quick mode: login, send cmds, and exit
  • 1.4. automate commands with user defined patterns
  • 1.5. "special" commands
    • 1.5.1. SLEEP N
    • 1.5.2. GOBACKGROUP
    • 1.5.3. REPEAT M N
    • 1.5.4. GRES
    • 1.5.5. MONITOR
  • 1.6. "projects"
  • 1.7. integration with shell script
    • 1.7.1. to pull data out of a device in shell
    • 1.7.2. to "scan" and pull data from all devices within a subnet:
    • 1.7.3. work with environment varible
  • 1.8. timestamp all your commands
  • 1.9. running crtc in background
    • 1.9.1. ctrl-g suspend current login session, make it running in background.
    • 1.9.2. some implementation details
  • 1.10. login multiple hosts simultaneously
    • 1.10.1. "parallel" mode
  • 1.11. "event script"
  • 1.12. "op script"
  • 1.13. "programmable" data capture
    • 1.13.1. example1
    • 1.13.2. example2
  • 1.14. hide login details
  • 1.15. persistent mode
  • 1.16. logs
  • 1.17. debug mode
  • 1.18. nested crtc
  • 1.19. command substitutions
  • 1.20. set "arbitrary" option: -a flag, and !a command
  • 1.21. built-in multi-party kibitz "on the fly"
    • 1.21.1. local sharing: sharing a terminal session with users in local server
    • 1.21.2. distributed sharing: sharing session with users in remote servers
  • 1.22. performance fine tune
  • 1.23. misc usages case
  • 1.24. more other working examples
• 2. usage cases
  • 2.1. device login
    • 2.1.1. login to any of your individual device:
    • 2.1.2. login to any router under a "jtac" class
    • 2.1.3. login to console of juniper jtac lab routers
    • 2.1.4. login to a customer’s device
  • 2.2. send commands
    • 2.2.1. -c option
    • 2.2.2. -e and -s options
  • 2.3. automate shell tasks
  • 2.4. file sync between juniper server and ATT server
    • 2.4.1. where is the script and how to run
    • 2.4.2. folders being sync.ed
    • 2.4.3. steps to copy a file from juniper server to att device
    • 2.4.4. steps to "upload" a file (coredump, logs, etc) from att device to juniper shell server
    • 2.4.5. the jtacsync.sh script
  • 2.5. scan all routers
  • 2.6. problem replication
  • 2.7. full parameterization
    • 2.7.1. compromise between config file and command line options
  • 2.8. capture a specific values
  • 2.9. timestamping all commands
    • 2.9.1. timestamp command from non-Junos devices
  • 2.10. features
  • 2.11. enable_user_patterns
  • 2.12. expect_matchany
  • 2.13. prefix_mark
  • 2.14. auto re-attemp with diff account
  • 2.15. share output to other terminal or a file
  • 2.16. semi-automation
  • 2.17. event script
  • 2.18. monitor device
• 3. introduction
  • 3.1. some design considerations
  • 3.2. feature highlights
• 4. how to run the script
  • 4.1. the script: just one file
  • 4.2. config file
  • 4.3. config options
• 5. crtc options
  • 5.1. options list
  • 5.2. option maps
  • 5.3. internal data structure (arrays)
    • 5.3.1. login_info
    • 5.3.2. cmdsN
    • 5.3.3. pre_cmdsN
    • 5.3.4. post_cmdsN
    • 5.3.5. regex_info
    • 5.3.6. issue_info
    • 5.3.7. collect
    • 5.3.8. testN
    • 5.3.9. an example
  • 5.4. user_patterns
• 6. more usage examples
  • 6.1. monitoring: keep sending cmd(s) in a loop
  • 6.2. group multiple options
  • 6.3. some "special" commands
    • 6.3.1. GRES command (Junos platform specific)
    • 6.3.2. "GRES" with "persistent" session
    • 6.3.3. repeat the "GRES" operation multiple iterations
    • 6.3.4. SLEEP command (platform-independent)
  • 6.4. integration with other tools in shell
    • 6.4.1. work with shell script
    • 6.4.2. work with GNU screen/tmux
  • 6.5. send log as email attachment
  • 6.6. other misc options
  • 6.7. activate features on the fly
  • 6.8. running crtc without options/parameters
    • 6.8.1. start new automations AFTER the initial automation finishs
    • 6.8.2. organize commands with nested command groups
  • 6.9. commands "on the fly" (keymaps)
    • 6.9.1. !t !T timestamp every commands
    • 6.9.2. !c!, !e!
    • 6.9.3. other key commands (t.b.c)
• 7. license
• 8. appendix
  • 8.1. add an option
  • 8.2. update notes and todo list
  • 8.3. some interesting references:
  • 8.4. test set (for testing crtc dev only)
  • 8.5. TODO and DONE
  • 8.6. known issue
• tcl notes
  • 9. tcl man pages
  • 10. tcl var
    • 10.1. ""
    • 10.2. {}
  • 11. string
    • 11.1. string replace
    • 11.2. string repeat
    • 11.3. string trim
    • 11.4. string index
    • 11.5. an empty string
    • 11.6. scan
    • 11.7. format
  • 12. list
    • 12.1. lsearch
    • 12.2. lrange
    • 12.3. llengh
    • 12.4. lindex
  • 13. array
  • 14. control structures
    • 14.1. switch
    • 14.2. -regexp -matchvar
  • 15. array operation
  • 16. string vs. list
    • 16.1. split and join: string vs list
    • 16.2. append vs. lappend
  • 17. list , concat ,eval
  • 18. backslash
  • 19. exec vs. system
  • 20. subst flags
    • 20.1. example
  • 21. dynamic coding: eval + subst
    • 21.1. example1
    • 21.2. example2
    • 21.3. cautions with subst
    • 21.4. cautions with eval
  • 22. continuation
  • 23. tcl comment
  • 24. tcl error processing: catch
  • 25. file operation
    • 25.1. file command
    • 25.2. open/read/flush/fconfigure
    • 25.3. pipeline: work with existing unix command!
    • 25.4. cd, pwd
  • 26. proc
    • 26.1. upvar: passing by reference (pointer)
    • 26.2. upvar vs global
    • 26.3. upvar to associate var1 var to var2 name
    • 26.4. uplevel
    • 26.5. unknown
  • 27. argv
  • 28. namespace
  • 29. regexp and regsub
  • 30. to kill a telnet/ssh session
  • 31. logical operation
    • 31.1. string compare
    • 31.2. ternary operator
  • 32. debugging
  • 33. cron/event-scheduler-like
  • 34. nested function call
  • 35. tcl error messages
  • 36. coding issues
  • 37. crontab
    • 37.1. error1: env
    • 37.2. error2: send_tty
    • 37.3. error3: stty
  • 38. misc
    • 38.1. editor/tcl shell/etc
    • 38.2. tclreadline
    • 38.3. teacup
    • 38.4. dejagnu
    • 38.5. make script executable but not readable ..
    • 38.6. return in source file
    • 38.7. NewHeadline
  • 39. reference
• expect notes
  • 40. expect resources
  • 41. expect
    • 41.1. expect syntax
    • 41.2. expect_out
    • 41.3. expect -indice
    • 41.4. expect -re
    • 41.5. expect -gl
    • 41.6. expect *
    • 41.7. expect (nothing)
    • 41.8. expect timeout
    • 41.9. expect eof
    • 41.10. expect pattern no-op
    • 41.11. why "\\\$"?
    • 41.12. glob vs. regex
    • 41.13. expect common patterns common example
    • 41.14. expect limitation and workaroud
    • 41.15. expect-send pair essentials
    • 41.16. expect vs regexp vs switch
    • 41.17. exp_continue
      • 41.17.1. absorbing extra chars
    • 41.18. expect -notransfer
    • 41.19. expect null
    • 41.20. expect_user
    • 41.21. expect -brace
      • 41.21.1. default "rule" of guess
      • 41.21.2. inconsistencies
      • 41.21.3. force a single argument treated as a single pattern
      • 41.21.4. force a single arugment treated as a list of patterns
    • 41.22. expect_before/expect_after
  • 42. send
    • 42.1. send vs. puts
  • 43. spawn
  • 44. email
  • 45. \r\n and stty mode
    • 45.1. all about \r\n
    • 45.2. raw/cooked mode
    • 45.3. mode changing: stty
  • 46. match_max and full_buffer
  • 47. log_user/log_file
    • 47.1. pending issue
  • 48. signal
  • 49. interact
    • 49.1. interact_out
    • 49.2. interact in a loop
    • 49.3. nested interact (interact under interact)
    • 49.4. interact default action: interpreter
    • 49.5. interact on eof
    • 49.6. interact timeout
    • 49.7. implicit spawn_id and -o
    • 49.8. -i (explicit spawn_id)
    • 49.9. -u
    • 49.10. -input -output
      • 49.10.1. kibitz
    • 49.11. -iwrite
    • 49.12. indirect spawn_id
    • 49.13. -reset
    • 49.14. puts under interact
    • 49.15. inter_return vs return
  • 50. expect vs. interact
    • 50.1. call expect_user under interact
    • 50.2. call expect under interact
    • 50.3. call interact under expect
    • 50.4. call interact under interact (nested interact)
  • 51. -i option
  • 52. Expect specific cmds
    • 52.1. interpreter
    • 52.2. prompt1/2
  • 53. Expect errors
  • 54. "screen" sharing
  • 55. debug Expect script
    • 55.1. debug 1
    • 55.2. trace
    • 55.3. exp_internal
    • 55.4. expect debugger
• tcl extentions
  • 56. sqlite: most popular tcl extension?

crtc is an expect script that can be used to automate the interactive tasks in a general manner:

• login to remote device(s)

• sending commands repeatedly

• issue/log/event monitoring

• monitor user-defined issues and execute user-defined actions

• shell friendly

• command (and output) timestamping

• logging

• misc:

  • anti-idle (keep alive)

  • switch the session in background/foreground,

  • online helping,

  • email notification,

  • etc

      ____ ____ _____ ____
     / ___|  _ \_   _/ ___|
    | |   | |_) || || |
    | |___|  _ < | || |___
     \____|_| \_\|_| \____|

1. quick examples

The script was created soly in "part-time" and just worked for my own usage case , the usage may not be fully tested yet due to the very limited time that I can find for this "project", therefore it’s very likely that a lot of functions may be still not fully working as expected here and there. But most of the usage examples listed in this doc were extracted from logs of my production work , so at least these are working fine when I recorded them :). I have tested the main part of the code and they are basically working, and I’m feeling they are even quite extendable that new function can be added easily, also small issues can be fixed relatively easy on demand. Just send me your improvement request and issue feedback.

Note	For Juniper folks without a machine with "Expect" tool installed, any of the svl-jtac servers (e.g. svl-jtac-tool01) will be sufficient. Try (copy and paste) following examples and have a quick overview of what it can do for you.

1.1. automate device login process

1.1.1. configure login info for individual device

• login to a new device named "myrouter"

  put this below login entry into ~/crtc.conf:

  #set domain_suffix_con jtac-west.jnpr.net
  #set domain_suffix jtac-east.jnpr.net
  set login "labroot"
  set password "lab123"

  #test only
  set login_info(myrouter)       [list                \
      "$"            "telnet 172.19.161.101"          \
      "login: "      $login                           \
      "Password:"    $password                        \
      ">"            "set cli screen-width 300"       \
      ">"            "set cli timestamp"              \
  ]

  then login to the device:

  ~pings/bin/crtc myrouter

1.1.2. configure login info for a group of device

• all devices under same "group" shares the same login steps. a group is indicated by appending a @GROUPNAME string after the device name. e.g.: all devices under suffix @jtac shares the same login steps.

  put this below login entry into ~/crtc.conf:

  set domain_east jtac-east.jnpr.net
  set domain_west jtac-west.jnpr.net
  set login_string "telnet $session.$domain_east"

  set login_info($session@jtac)       [list       \
      "\\\$"        "$login_string"    \
      "login: "      "$jtaclab_login"             \
      "Password:"    "$jtaclab_pass"              \
      ">"            "set cli screen-width 300"   \
      ">"            "set cli timestamp"          \
  ]

  then login to the device:

  ~pings/bin/crtc alecto@jtac

1.2. host name resolution

• having these in crtc.conf:

  array set hostmap {\
       pe5        172.19.161.107 \
       pe6        192.168.43.17\
  }

  set login_info($session)       [list       \
      "\\\$"        "$host"    \
      "login: "      "labroot"             \
      "Password:"    "lab123"              \
      ">"            "set cli screen-width 300"   \
      ">"            "set cli timestamp"          \
  ]

  the "session" name pe5 will be resolved to a "host" IP. now you can login pe5 with:

  crtc pe5

• parsing the user-defined host string for login info

  with below configuration, script first resolve session name "alecot-re0-con" into string b6tsb17:7021, then parse it to get terminal server name and telnet port. this is how the script login to the "console" of a router:

  array set hostmap {\
      alecto-re0-con	 b6tsb17:7021	\
      alecto-re1-con	 b6tsb17:7022	\
      havlar-re0-con	 b6tsb17:7034	\
  }

  if [regexp {^(\w+):(\d+)} $host -> hostname port] {

  #b6tsb25:7028
  set fullname "$hostname.$domain_west"
  set login_string "telnet $fullname $port"

  }

  set login_info($session@jtac)       [list       \
      "\\\$"        "$login_string"    \
      "login: "      "$jtaclab_login"             \
      "Password:"    "$jtaclab_pass"              \
      ">"            "set cli screen-width 300"   \
      ">"            "set cli timestamp"          \
  ]

  login to console port of router alecto:

  ~pings/bin/crtc alecto-re0-con@jtac

1.3. automate commands after login

• run a command (-c) 10 times (-n), with interval of 2 seconds (-i), don’t display (hide) the login steps (-H), and quit the session when all done (-q)

  ~pings/bin/crtc -c "show chassis alarms" -H -q -n 10 -i 2 myrouter

  group all options to make it a little bit shorter:

  ~pings/bin/crtc -Hqi2n10c "show chassis alarms" myrouter

• flap a port 3 times with 10s interval, in each iteration: shutdown and wait 3 seconds, then bring up (rollback)

  ~pings/bin/crtc -b "configure" \
    -c "set interfaces xe-3/1/0 disable" \
    -c "commit" -c "SLEEP 3" -c "rollback 1" \
    -c "show | compare" -c "commit" \
    -B "exit" -n 3 -i 10 myrouter

  or, dropping into shell and use shell command to do the same:

  crtc -E ">" -S "start shell" -E "%" \
    -S "su" -E "sword" -S "Juniper" \
    -c "ifconfig xe-3/1/0 down"  \
    -c "SLEEP 3"                 \
    -c "ifconfig xe-3/1/0 up"    \
    -n 3 -i 10 -g 5 myrouter

  -g is interval between each command during the same iteration.

  this maybe look too long, the same thing can be done with following options configured in ~/crtc.conf ^[1].

  set pre_cmds1(myrouter) configure
  set cmds1(myrouter) {
      "set interfaces xe-3/1/0 disable"
      commit "SLEEP 3" "rollback 1"
      "show | compare" commit
  }
  set post_cmds1(myrouter) exit

  then just run crtc without options:

  ~pings/bin/crtc -n3j myrouter

  here -j is to specify a "project", which simply maps to the number of cmds array that holds the actual commands to be executed. in this case we want to execute commands in cmds1, to "-j" or "-j 1" will do it.

1.3.1. interupting an automation loop

Instead of just awaiting for sleep time to expire, crtc provides more options for the user to interact with the script, even during the sleep time:

• press + to increase sleep interval by 2s

• press - to decrease sleep interval by 2s

• press <space>, <ENTER> , or any other key to skip the current sleep time and start next iteration right away.

• press q to "pause" the iteration and return the session control to the user, who can continue to type commands, and :

  • press !R to resume the iteration from wherever left off (before the iteration was interupted by pressing q)

  • press !s to stop the automation

  • press !r to start the automation all over again

This commands help a user to have better control in a session to start/pause/resume/restart a pre-defined automation.

1.3.2. quick mode: login, send cmds, and exit

-q option:

this is useful for quick test in shell script, or you just need to get some quick data in one shot. Imaging that in the middle of your shell script you need some (realtime) data from the router, calling this script you can login to a router, send the commands and grab the data, after that you need the crtc to quit so the original shell script can continue from where it left off …​ the -q option is used to just "quit-after-done".

config

this is equivalent to having this settings in config file:

set nointeract 1

Example 1. login with -q

ping@ubuntu1404:~/bin$ crtc -c "show system uptime" -H -q alecto
current log file ~/logs/alecto.log
set cli timestamp

Dec 01 13:14:24
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0> bye!:)
ping@ubuntu1404:~/bin$

Tip	later I’ll provide examples about how to use crtc in your shell script.

1.4. automate commands with user defined patterns

adding below patterns in crtc.conf:

set user_patterns(pattern_not_resolve_msg)              [list "ould not resolve"]
set user_patterns(pattern_connection_unable)            [list "telnet: Unable to connect to remote host"]
set user_patterns(pattern_console_msg)                  [list "Type the hot key to suspend the connection: <CTRL>Z"]
#set user_patterns(pattern_connection_not_responding)    [list {Timeout, server (\\d{1,3}\\.){3}\\d{1,3} not responding}]
set user_patterns(pattern_connection_not_responding)    [list {Timeout, server (\d{1,3}\.){3}\d{1,3} not responding}]
set user_patterns(pattern_broken_pipe)                  [list "Write failed: Broken pipe"]
set user_patterns(pattern_connection_close_msg)         [list                                                                   \
    {Connection closed.*|Connection to (\d{1,3}\.){3}\d{1,3} closed|Connection to \S+ closed|Connection reset by peer}  \
    RECONNECT]
set user_patterns(connection_refused)                   [list "Connection refused" RECONNECT]

set user_patterns(pattern_gres_not_ready)               [list {Not ready for mastership switch, try after \\d+ secs.*}]

set user_patterns(answer_yes)                           [list {\(no\)} yes]
set user_patterns(backup_re_not_running)                [list "error: Backup RE not running" RETRY]
set user_patterns(backup_re_not_ready)                  [list "error: Backup RE not ready for ISSU" RETRY]
set user_patterns(nsr_not_active)                       [list "warning: NSR not active" RETRY]
set user_patterns(issu_aborted)                         [list "error: ISSU Aborted" RETRY]

now to iterate JUNOS "ISSU" 10 time, use this command:

crtc -n10pc "request system software in-service-upgrade /var/tmp/xxxxx reboot" getafix@jtac

to make it short and less ugly, also add this in crtc.conf:

set issu_image_debug    "junos-install-mx-x86-64-15.1I20160311_0949_abhinavt.tgz"
set issu_image_s51      "junos-install-mx-x86-64-15.1F2-S5.1.tgz"
set issu_cmd_debug      "request system software in-service-upgrade /var/tmp/$issu_image_debug reboot"
set issu_cmd_s51        "request system software in-service-upgrade /var/tmp/$issu_image_s51 reboot"

set cmds1(pe50@attlab) [list   \
    "$issu_cmd_debug"        \
]

#new method:with user_patterns, it can be simplified as below:
set ISSU(getafix@jtac) [list   \
    "$issu_cmd_s51"        \
]

now a JUNOS "ISSU" automation can be done by this better-looking command:

crtc -n10pc ISSU getafix@jtac

how it works?

the -p indicates a "persistent mode". Under this mode crtc will use one of the configured user_pattern to detect connection loss:

set user_patterns(pattern_connection_close_msg)         [list                                                                   \
    {Connection closed.*|Connection to (\d{1,3}\.){3}\d{1,3} closed|Connection to \S+ closed|Connection reset by peer}  \
    RECONNECT]

once a match is detected, the user configured action will be executed. In this case it’s to "RECONNECT", so crtc will try to reconnect to the router and continue the rest of the iterations.

1.5. "special" commands

1.5.1. SLEEP N

• example of using config file

  add following in in the configuration file: ~/crtc.conf

  set cmds1(myrouter)         [list \
      "show system uptime"        \
      "SLEEP 15"                  \
      "show chassis alarm"        \
      "SLEEP 30"                  \
      "show system processes extensive | no-more"        \
  ]

  run the script:

  ~pings/bin/crtc -n 3 -i 5 myrouter

  this will make crtc to login the router, and repeat the following command set 3 times:

  • execute "show system uptime"

  • sleep for 15s

  • execute "show chassis alarm"

  • sleep for 30s

  • execute "show system processes extensive | no-more"

• same,but using CLI options without bothering any config file

  without bothering to change config file, you can use entirely CLI options instead. In this example we login to router named "myrouter", send "show system uptime" 3 times with a 5s interval

  ~pings/bin/crtc -c "show system uptime" -c "SLEEP 15"        \
                  -c "show chassis alarm" -c "SLEEP 30"        \
                  -c "show system processes extensive | no-more" \
                  -n 3 -i 5 myrouter

  Note	you can also mix CLI flags with config file options - leave the login info in the config file, but use CLI flags to send commands

Tip	if value of seconds is not given, it will sleep 3s.

another example:

set cmds2(pe50@attlab) [list   \
    ">" "request routing-engine login other-routing-engine" \
    ">" "start shell"   \
    "%" "su"            \
    "sword" "jnpr123"   \
    "#" "SLEEP 5;dtrace -s /var/tmp/chassisd_socket.d -o chassisd_snmpd_%T.log&" \
    "#" "exit"          \
    "%" "exit"          \
    ">" "exit"          \
    ">" "SLEEP 5"       \
    ">" "GRES"              \
]

1.5.2. GOBACKGROUP

As the name indicates, this command can be used to bring a task to run in "background". This is similiar to the linux "task management" tools in shell that is useful when multiple processes (e.g. one per router) need to be run at the same time. one practical example is shown below:

1. first, configure the router login process to login the router and drop into the shell

   set login_info(automatix_shell@jtac)       [list       \        (1)
       "\\\$"      "crtc -d0 automatix@jtac"    \                  (2)
       "automatix@jtac:automation done"    "start shell"        \  (3)
       "%"         "su"                    \                       (4)
       "sword:"    "Juniper"               \                       (4)
       "$"         "uptime"                \                       (4)
   ]
   
   set login_info(dogmatix_shell@jtac)       [list       \         (1)
       "\\\$"      "crtc dogmatix@jtac"    \                       (2)
       "dogmatix@jtac:automation done"    "start shell"        \   (3)
       "%"         "su"                    \                       (4)
       "sword:"    "Juniper"               \                       (4)
       "$"         "uptime"                \                       (4)
   ]

   1. steps to login to a JUNOS router’s shell

   2. use recursive crtc to login to a router

   3. once the child crtc done its job, enter JUNOS shell

   4. login with su

2. second, config the process to "pre-build" both router sessions, and move them into background so they can be used as needed later

   set login_info(LOCALHOST) [list \
       "\\\$"      "uptime"          \                         (1)
       "\\\$"      "crtc -d0 automatix_shell@jtac" \           (2)
       "automatix_shell@jtac:automation done" "uptime" \       (3)
       "%"    "GOBACKGROUND"  \                                (4)
       "\\\$"      "crtc -d0 dogmatix_shell@jtac" \            (5)
       "dogmatix_shell@jtac:automation done" "uptime" \        (6)
       "%"    "GOBACKGROUND"  \                                (7)
       "\\\$" "jobs"       \                                   (8)
   ]

   1. crtc start with printing a time from local shell (where crtc is running)

   2. crtc invokes a "nested" or "child" crtc process crtc -d0 automatix_shell@jtac, this will login to the shell of router automatix

   3. crtc monitor the progress of the child crtc process, when it’s done, run a uptime command to get another timestamp, this time from the remote router shell.

   4. crtc execute "GOBACKGROUND" command, which will send a signal to put the child crtc process to background. this will release the current terminal back to crtc, so other tasks can be performed.

   5. once the prompt from original terminal appears, crtc invokes another child process to login to the shell of another router dogmatix

   6. same as in <3>, crtc monitor the progress of the router login process, and acquire another timestamp from the shell of the new router.

   7. same as in <4>, crtc move this new child crtc process to background

   8. crtc now has 2 child processes, each representing a session into a remote router, running in background

3. config the cmdN array to do the test on the two routers

   set cmds9(LOCALHOST) [list                              \
       "fg 1\r\n"                                          \       (1)
       "uptime"                                            \       (2)
       "ifconfig xe-0/2/0 down;ifconfig xe-0/2/0 up"       \       (3)
       "SLEEP 6"                                           \       (4)
       "ifconfig xe-0/2/0 down;ifconfig xe-0/2/0 up"       \       (5)
       "SLEEP 300"                                           \     (6)
       "GOBACKGROUND"                                      \       (7)
       "fg 2\r\n"                                          \       (8)
       "uptime"                                            \       (9)
       "gzip -f capture1.txt"                              \       (9)
       {cprod -A fpc0 -c "show nhdb summary"}              \       (9)
       {echo "check fpc0" >> capture1.txt}                 \       (9)
       {grep -E "Inactive|Uninstall" capture1.txt | wc -l} \       (9)
       "GOBACKGROUND"                                      \       (10)
   ]

   1. get router automatix session: move the session (first session) foreground

   2. capture a timestamp from this router

   3. flap link xe-0/2/0 of automatix

   4. sleep 6s

   5. flap same link again

   6. sleep 300s

   7. move the router automatix session to background

   8. get router dogmatix session: move the session (2nd sesson) foreground

   9. get a timestamp from this router, and perform other test steps

   10. move this router session to background

4. finally, to iterate the test 10 times:

   ping@ubuntu47-3:~$ crtc -n10 LOCALHOST

1.5.3. REPEAT M N

considering a test case when it is required to repeat some commands several times during one iteration, e.g:

set cmds1(myrouter) [list                    \
    "GRES"                                   \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
]

in this configuration, we need to iterate these test in sequence:

1. perform GRES

2. check if there is any BGP flap

3. wait 10s

4. repeat previous 2 steps 5 more times

The goal is to iterate this whole process 50 times, and in each iteration we need to check the number of BGP connections every 10s during at least a minute long. the reason we want to "repeat" the check is - so we won’t miss a BGP flap that whould happen to occur randomly in 1 minute after GRES performed, but then recovered quickly. repeating the same check on the number of bgp sessions will ensure that we’ll catch the issue whenver it happened. while this works fine, the config looks ugly - it’s even ugly if we need to repeat 200 more times in other cases.

A better way is to use the REPEAT command, so the above config can be rewritten to:

set cmds1(myrouter) [list                    \
    "GRES"                     \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
    "REPEAT 2 5"                             \
]

the command REPEAT 2 5, just means to repeat the previous whatever 2 commands, 5 more times.

for completeness here are the complete configs (in config file crtc.conf) for this specific test case:

set cmds1(myrouter) [list                    \ (1)
    "GRES"                                   \
    "show bgp summary | match estab | count" \
    "SLEEP 10"                                \
    "REPEAT 2 5"                             \
]

set regex_info(myrouter) {                     (2)
    {2@@Count: (\d+) lines@bgpcount}
}
#Count: 0 lines

set issue_info(myrouter) {                     (3)
    {2@bgpcount != 2}
}

set collect(myrouter) [list      \             (4)
    "show bgp summary | no-more" \
    "show log message | match bgp | last 20"
]

1. perform RE switchover, then repeatedly check bgp connection numbers at least 6 times in a minute, before moving to the next iteration

2. with a regex, capture the number of established BGP sessions using the number 2 command ("show bgp …​") , save it to a varible named bgpcount

3. define the "issue" to be that whenever the value of varible bgpcount is NOT 2

4. if the "issue" is "seen", collect some more data

and this is how the crtc runs:

crtc -n50j myrouter

this will:

• iterate the same above test procedure 50 times (-n50)

• use commands defined in project 1 (array cmds1)

1.5.4. GRES

this command:

~pings/bin/crtc -c "GRES" -c "show chassis alarm" -Hqn30i250 myrouter

will do:

• perform JUNOS "RESO" 30 times (-n30)

• and quit(-q) after all done

• it will wait 250s between each RESO (-i250)

• suppress the detailed output (-H)

skipping -i250 is fine, "GRES" command has built-in "visibility" to read the JUNOS output, and delay the next RESO attempt accordingly. see more detail in section GRES

Note	240s is the minimum waiting time between 2 RESO in Juniper router.

1.5.5. MONITOR

TODO

1.6. "projects"

As you keep adding more data and changing the commands for different cases in the config file crtc.conf, it will grow bigger and contains a lot of different, and often conflicting command groups with each of them serving a specific test case you ever worked on. use -j to specify which command group you prefer crtc to send, here is an example, in crtc.conf you have below configured command groups:

• cmds1

• cmds2

• cmds6

set cmds1(myrouter) {
    "show system core"
    "show chassis alarm"
}

set cmds2(myrouter) {
    "show version | no-more"
    "show chassis fpc pic-status"
    "show chassis hardware | no-more"
}

set cmds6(myrouter) {
    "systeminfo"
    "ospfinfo"
    "bgpinfo"
}

set systeminfo(myrouter) {
    "show system uptime"
    "show chassis alarm"
}

set ospfinfo(myrouter) {
    "show ospf neighbor"
    "show ospf overview"
}

set bgpinfo(myrouter) {
    "show bgp summary | match up"
}

to execute commands defined in cmds2, run this:

crtc -j2 myrouter

-j is used to specific a "project" number, in this case it is 2, so cmds2 will be executed. All other command groups can remain in the config file in case they are needed for other tests.

1.7. integration with shell script

1.7.1. to pull data out of a device in shell

copy and paste these 2 lines in the shell server:

ver=`crtc -HWqc "show version | no-more" myrouter | grep -i "base os boot" | awk '{print \$5}'`
echo "myrouter is running software version: $ver"

+ you’ll get:

myrouter is running software version: [12.3R3-S4.7]

+ this will print the current version that router alecto is running. the -W means "use crtc in shell", so all messages printed by crtc will now be "pipe-friendly" - without this option some messages will be just printed to the terminal regardless of whether crtc is running in another shell script or not.

• To pull running version from any junos device

  put the 2 lines in a file, name it printver.sh, in jtac server:

  #file: printver.sh
  #!/bin/bash
  ver=`~pings/bin/crtc -HWqc "show version | no-more" $1 | grep -i "base os boot" | awk '{print \$5}'`
  echo "router $1 is running software version: $ver"

  then run the shell script with a router name as the only parameter ^[2]. It will detect and report the current release info from any given Junos router :

  JTAC lab router:

  [pings@svl-jtac-tool02 ~]$ ./printver.sh tintin@jtac
  router tintin@jtac is running software version: [12.3R3-S4.7]

  customer router(@att needs to be configured in config file):

  [pings@svl-jtac-tool02 ~]$ sh printver.sh DTxxxxVPE@att
  router DTxxxxVPE@att is running software version: [14.1-20141106.0]

1.7.2. to "scan" and pull data from all devices within a subnet:

e.g. to pull hardware inventory data from whole subnet 172.19.161.x:

for i in {2..254}; do crtc -HW5w5qac "show chassis hardware" 172.19.161.$i@jtac; done;

+ if only to pull info from specific IP in the subnet:

for i in 2 3 5; do crtc -HW5w5qac "show chassis hardware" 172.19.161.$i@jtac; done;

+ or, pull info from a group of seperated IP addresses that are not in same subnet:

for i in 1.1.1.1 2.2.2.2 3.3.3.3; do crtc -HWw3qac "show chassis hardware" -w 5 $i@jtac; done;

+

• the -A flag is used to automatically "page" the long output from the command, which will otherwise requires: | no-more;

• -H hide the login steps,

• -q make the crtc quit right after data collected. without this crtc will go into interact mode on first session , as a result the shell script will stay in first router session and won’t proceed to the next router until you manually exit each session.

• -w5W5 to wait maximum 3s before timeout current session, this is to make crtc exit timely if the remote IP is a "dead" peer, so the shell script will proceed anyway.

+ NOTE: in practice, the list of IP can go very much longer than these. a reasonable requests will be to pull data periodically from 50 to 500 routers in a network, for monitoring purpose. in that case a better practice is to put this long-one-liner into a file, and execute it as a shell script.

1.7.3. work with environment varible

suppose we want to use the login steps configured in crtc.conf, like in configure login info for individual device , but we want to change the value of varible login and password to sth else. one way is to just change the value in crtc.conf:

set login "labroot1"
set password "lab456"

another way is to change it directly from shell:

ping@ubuntu47-3:~$ export CRTC_login=labroot1
ping@ubuntu47-3:~$ export CRTC_password=lab456

now crtc will login the router with the changed login and password.

ping@ubuntu47-3:~$ crtc myrouter

to sKip the environment variable and just force using the original varible defined in config file, use -K:

ping@ubuntu47-3:~$ crtc -K labrouter@attlab

1.8. timestamp all your commands

• to timestamp all your unix, Junos shell, PFE command commands

lab@mx86-jtac-lab> start shell
Dec 03 08:20:30                     (1)
% ls -l /var/log/messages           (2)
-rw-rw----  1 root  wheel  4194094 Dec  3 08:21 /var/log/messages

%!timestamp 1 - local timestamp on! (3)

Dec 03 05:27:37 2014(local)
% ls -l /var/log/messages
Dec 03 05:27:47 2014(local)         (4)
-rw-rw----  1 root  wheel  4195050 Dec  3 08:25 /var/log/messages

root@mx86-jtac-lab% vty 1
Dec 03 05:28:17 2014(local)

BSD platform (Pentium processor, 1536MB memory, 0KB flash)

VMXE(mx86-jtac-lab vty)# show pfe statistics traffic
Dec 03 05:28:35 2014(local)         (5)
PFE Traffic statistics:
                    0 packets input  (0 packets/sec)
                    0 packets output (0 packets/sec)

PFE Local Traffic statistics:
...<snipped>...

1. the last timestamp provided by Junos "set cli timestamp"

2. no timestamp provided in native unix shell mode

3. you press !t: and crtc will prompt a local timestamp will be provided for each cmd you typed, also crtc prompted that timestamp option is set

4. shell commands now got timestamped

5. pfe commands now got timestamped

   • to timestamp e320 shell command

     [pings@svl-jtac-tool02 ~]$ ~pings/bin/crtc -Ht e320-svl
     ...<snipped>...
     slot 16->print__11Ic1Detector       #<------Junos-e vxWorks shell command
     Dec 02 15:25:14 2014(local)         #<------"local" timestamp by crtc

     state=NORMAL
     sysUpTime=0x03B33619
     passiveLoader=0x0C001994
     crashPusherRequested=1
     ...<snippet>...

1.9. running crtc in background

Very often you want to logging into different remote devices to check the network issue. your current temrinal will be occupied by the first crtc session after you login into one remote device. In order to to login to the 2nd or even more devices in the same terminal, there must be a way to "hang up" current job, start the 2nd session, work on the 2nd session when the 1st session is still running in the background.

Note	other common practices to solve this is to use multiple terminals - either to open multiple terminal windows or open multiple terminal "tabs" in one window.

There are multiple options of doing this:

• running crtc within GNU screen (so you can shutdown your own PC and leave, while the session keeps running in the remote server) + first login to a server where GNU screen is available:

  ssh svl-jtac-tool02

  then start crtc "within" a GNU screen window:

  [pings@svl-jtac-tool02 ~]$ screen -fn -t myrouter ~pings/bin/crtc myrouter

  now the session to myrouter is "held" by a screen window named myrouter, which can be running independently with your client PC. shutting down client PC won’t affact anything to the running script and it will just keep running behind the scene. This should look familiar to GNU screen user because it is the most typical usage scenario of the tool.

• use Expect dislocate tool

  ping@ubuntu1404:~$ dislocate crtc -H myrouter

  press ctrl-] then type ctrl-D or exit to "detach" the session:

  lab@alecto-re1>
  to disconnect, enter: exit (or ^D)
  to suspend, press appropriate job control sequence
  to return to process, enter: return
  /usr/bin/dislocate1> exit
  ping@ubuntu1404:~$

  same as in the case of GNU screen, crtc script is still running in the background even if it is not shown in the current terminal. whenever you need it back, run dislocate again to connect the session back

  ping@ubuntu1404:~$ dislocate
  connectable processes:
   #   pid      date started      process
   1  29571  Sat Apr 11 11:11:28  crtc -H myrouter
   2  27225  Sat Apr 11 10:29:56  crtc rams@jtac
  enter # or pid: 1
  Escape sequence is ^]

  Apr 11 11:12:12

  lab@alecto-re1>

• move current session to run in background, just press ctrl-g any time during the session

1.9.1. ctrl-g suspend current login session, make it running in background.

With crtc, afer logged into the router successfully, anytime during the session, the current session can be suspended by pressing ctrl-g .

Note	This is not possible if the session was established using the default telnet client.

{master}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re0> set cli timestamp

Nov 22 12:51:05
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0> [Sat Nov 22 12:50:33 EST 2014::..c-g was pressed!..]

[1]+  Stopped                 crtc alecto@jtac
ping@ubuntu1404:~$ ^C

to continue, fg it to run in the front.

ping@ubuntu1404:~$ fg
crtc alecto@jtac

Nov 22 12:52:55

{master}
lab@alecto-re0>

This make it easier to manage multiple active sessions in the same terminal at the same time, without using external tools like GNU screen/tmux/etc.

1.9.2. some implementation details

crtc will "monitor" every keystroke from a user and execute the implemented actions once the associated key was "seen" by crtc. currently, by default press ctrl-g will trigger a "hangup" system signal to current script, which will effectively move it running "background". It can then be killed (by sending a SIGKILL signal) as a normal process with the traditional unix kill command .

Example 2. ctrl-g to move the script running in background

{backup}
lab@alecto-re0> c-g was pressed, move this session background!

[1]+  Stopped                 crtc alecto-re0-con@jtac
ping@ubuntu1404:~$ kill -9 %1

[1]+  Stopped                 crtc alecto-re0-con@jtac
ping@ubuntu1404:~$
[1]+  Killed                  crtc alecto-re0-con@jtac
ping@ubuntu1404:~$

the ctrl-g key make the script behaving in a "compatible" manner within unix environment. traditionally in many telnet/ssh clients, (most of) keystrokes will be sent uninterpretedly to the remote device, instead of being intercepted locally. This is useful in that you can send control characters to trigger some actions to the processes in the remote device that you are working on. But that makes it hard to suspend the local client. While we can continue this behavior in crtc script, it will be handy to make use of some not-so-useful keystroke to do a useful job from the client - that is why ctrl-g is choosen to trigger a hang-up of the client, this is configurable from config file though.

Example 3. moving crtc backgound/forground just like a normal apps

with ctrl-g it is easy to run multiple crtc instance to login to different routers in the same terminal.

ping@ubuntu1404:~/bin$ crtc -H alecto
current log file ~/logs/alecto.log
it's all yours now!:)
set cli timestamp

Dec 02 14:07:28
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re1>
Dec 02 14:07:28

{master}
lab@alecto-re1> c-z was pressed, move this session background!

[1]+  Stopped                 crtc -H alecto
ping@ubuntu1404:~/bin$ crtc tintin@jtac
current log file ~/logs/tintin.log
telnet tintin.jtac-east.jnpr.net
ping@ubuntu1404:~/bin$ telnet tintin.jtac-east.jnpr.net
Trying 172.19.161.18...
Connected to tintin.jtac-east.jnpr.net.
Escape character is '^]'.
 Warning Notice

Please contact Pratima or Brian before making any changes

tintin-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3R3-S4.7 built 2014-06-30 05:41:54 UTC

lab@tintin-re0> set cli screen-width 300
Screen width set to 300

lab@tintin-re0> it's all yours now!:)
set cli timestamp

Dec 02 14:07:49
CLI timestamp set to: %b %d %T

lab@tintin-re0>
Dec 02 14:07:49

lab@tintin-re0> c-z was pressed, move this session background!

[2]+  Stopped                 crtc tintin@jtac
ping@ubuntu1404:~/bin$ jobs
[1]-  Stopped                 crtc -H alecto
[2]+  Stopped                 crtc tintin@jtac

now you can use the standard unix job control commands to select which task (session) you want to work on:

go to alecto session:

ping@ubuntu1404:~/bin$ fg 1
crtc -H alecto

Dec 02 14:08:53

{master}
lab@alecto-re1>
Dec 02 14:08:55

{master}
c-z was pressed, move this session background!

[1]+  Stopped                 crtc -H alecto

go to tintin session:

ping@ubuntu1404:~/bin$ fg 2
crtc tintin@jtac

Dec 02 14:08:59

lab@tintin-re0>
Dec 02 14:08:59

lab@tintin-re0>

1.10. login multiple hosts simultaneously

following command will login to 3 routers all together.

crtc -h router1 router2 router3

with shell expansion it can be shortened as:

crtc -h router{1..3}

to switch to the 1st session press \1, to go to 2nd session press \2, and so on. \i will print info of "current" host.

send same command(s) multiple times(-n 3 -i 5)to all routers at the same time (-P):

crtc -n3i5Pc "show system uptime" -h alecto@jtac automatix@jtac

this "multi-hosts" feature is currently only experimental and not well implemented yet (no much usage case)

1.10.1. "parallel" mode

TODO

1.11. "event script"

to emulate an "Junos event script", add these configurations in crtc.conf:

#event script example
set event1 "LINK_DOWN"
set event2 "LINK_UP"
set action1(myrouter)            {
    "show system uptime"
    "#link down detected!"
}
set action2(myrouter)            {
    "show system uptime"
    "#link up detected!"
}

set eventscript(myrouter)       [list       \
    "$event1"        "action1"              \
    "$event2"        "action2"              \
]

now run crtc to monitor the configured events:

crtc -Jc "monitor start messages" myrouter

This will:

• monitor syslog messages for event "LINK_DOWN" and "LINK_UP"

• when "LINK_DOWN" event is "seen", commands configured in action1 will be executed.

• when "LINK_UP" event is "seen", commands configured in action2 will be executed.

1.12. "op script"

to emulate an Junos "op script", add these configs in crtc.conf:

set cmds6(myrouter) {
    "systeminfo"
    "ospfinfo"
    "bgpinfo"
}

set systeminfo(myrouter) {
    "show system uptime"
    "show chassis alarm"
}

set ospfinfo(myrouter) {
    "show ospf neighbor"
    "show ospf overview"
}

set bgpinfo(myrouter) {
    "show bgp summary | match up"
}

This is to define some new "commands":

• systeminfo

• ospfinfo

• bgpinfo

Now these commands can be refered within a session, and crtc will "resolve" each of them into the real, target commands.

To refer these newly defined commands, use the !c "inline" command to invoke interactive question&answers, which allows you to input the defined commands in a session:

labroot@alecto-re0> !c      #<------
select your choice below:
1: enter command, or command array(cmds_cli)
   configured in config file to be executed
q: quit
1                           #<------
Enter the command/command array you want to iterate:
   []
ospfinfo                    #<------

Enter how many iterations you want to run: [1]
2                           #<------
Enter intervals between each iteration:  [0]
3                           #<------
will iterate command/command group [ospfinfo]  2 rounds with interval 3
    between each iteration,(y)es/(n)o/(q)uit?[y]
cmds_c(myrouter) = ospfinfo

<<<< start the iterations (2 rounds)

`- - - - - - - - - -  iteration:1  - - - - - - - - - - `
<<<<[iteration:1]=> myrouter:
<<<<  ospfinfo

labroot@alecto-re0> show ospf neighbor
Feb 16 23:07:45
Address          Interface              State     ID               Pri  Dead
10.192.0.41      xe-3/1/0.0             Full      192.168.0.6      128    36
10.192.0.45      xe-4/1/0.0             Full      192.168.0.7      128    38
1.1.1.2          so-1/2/0.0             Full      100.100.100.100  128    33
2.2.2.2          so-1/2/1.0             Full      20.20.20.20      128    36

labroot@alecto-re0> show ospf overview
Feb 16 23:07:45
Instance: master
  Router ID: 192.168.0.69
  ...<snippet>...

labroot@alecto-re0>
<<<<count {3}s before proceeding...
<<<<type anything to skip...

`- - - - - - - - - -  iteration:2  - - - - - - - - - - `
<<<<[iteration:2]=> myrouter:
<<<<  ospfinfo

labroot@alecto-re0> show ospf neighbor
show ospf overview
Feb 16 23:07:48
Address          Interface              State     ID               Pri  Dead
10.192.0.41      xe-3/1/0.0             Full      192.168.0.6      128    33
10.192.0.45      xe-4/1/0.0             Full      192.168.0.7      128    35
1.1.1.2          so-1/2/0.0             Full      100.100.100.100  128    39
2.2.2.2          so-1/2/1.0             Full      20.20.20.20      128    33

labroot@alecto-re0> show ospf overview
Feb 16 23:07:48
Instance: master
  Router ID: 192.168.0.69
  Route table index: 0
  ...<snippet>...

labroot@alecto-re0>
<<<<count {3}s before proceeding...
<<<<type anything to skip...
all done!
labroot@alecto-re0>

Note	the command definition can be "nested". you can define a command named routinginfo, which refers to ospfinfo, isisinfo and bgpinfo, each of which can either refers to the real junos commands, or other commands you wish to define in the crtc.conf file.

1.13. "programmable" data capture

1.13.1. example1

in below config:

set login_info(vmx-avpn.riot@jtac)       [list     \
    "$"      "crtc -vHQ0 vmx-avpn.vpfe@jtac"       \
    "vmx-avpn.vpfe@jtac:automation done"    "su"   \
    "sword:" "root"                                \
    "#"      "$riot_stats_logging"                 \
    "Logs are stored in (.+)\r" "cat %1"           \
]

the last pattern-action pair:

"Logs are stored in (.+)\r" "cat %1"           \

what it does is, to capture regex:

"Logs are stored in (.+)\r"

from the previous command output, and use %1 to refer the "back-reference" to the data captured with the 1st '()'. the next command is then composed based on this referenced data and sent to the device. this can be conveniently used to program the command based on the previous command output, dynamically.

1.13.2. example2

Note	this should work, but with a much more complicated method, not fully tested yet. will develop more on demand.

with below arrays defined in crtc.conf:

set cmds2(myrouter) [list                       \
    "show ospf neighbor"                        \
    "show interface %xe310 terse"               \
    "ping 10.192.0.108 count 2 source %ip"      \
]

set regex_info(myrouter) {
    {1@@(\S+)\s+Full@xe310}
    {2@@inet\s+(\S+)/30@ip}
    {3@@(\s+) packet loss)@percentage}
}

set issue_info(myrouter) {
    {3@percentage=="100%"}
}

when running crtc as below:

crtc -n10j2 myrouter

it will run each command in sequence, and it will use the data captured in previous command, to substitute the corresponding variables specified in the subsequent commands, creating a programmable data capture command list.

this is how it works:

1. j2 specifies project 2, so data defined in cmd array cmds2 will be read and executed

2. the first cmd show ospf neighbor is executed first

3. from the output of the first cmd, crtc will scan and capture the neighbor interface name, using a user defined regex (\S+)\s+Full

4. if anything got captured, the value will be saved into a user provided variable named xe310

5. crtc continue to run the 2nd cmd show interface %xe310 terse

6. the % indicate a varible evaluation will happen - crtc will evaluate the value of variable xe310, and substitute this varible with it’s value captured in previous cmd, in this case the value will be xe-3/1/0.0, then the final cmd show interface xe-3/1/0.0 terse will be sent to the router

7. this process will continue: from the output of the show interface .. command, crtc scan the output and try to capture and IPv4 IP address, using regex inet\s+(\S+)/30 defined in regex_info array. the captured value, if any, will be saved into the variable named ip, for later references

8. the last cmd is ping 10.192.0.108 source %ip, again, the %ip will be substituted by the value of variable ip, in this case for example it’s 10.192.0.42 so the real cmd sent to the router will be ping 10.192.0.108 source 10.192.0.42

9. from the output crtc will search for the percentage of packet loss, and then use that to determine if "a problem" occurs, actions can then be defined (in collect array).

1.14. hide login details

It’s useful to have all login details printed out if the login was made manually - the telnet/ssh client will print out the username/password prompt and wait for your input.

there are some other conditions that it may be better to "hide" the login process :

1. the use of a script to automate the whole process make these messages and username/password interactions useless.

2. In some cases it’s more secure to not even display the username as well as password.

3. the output will look neat if login process was skipped.

the display of login process can be suppressed by setting hideinfo option in crtc.conf file:

set hideinfo 1

same effect can be achieved by using -H flag in command line.

Example 4. login with -H option:

ping@ubuntu1404:~$ crtc -c "show system uptime" -H alecto
current log file ~/att-lab-logs/alecto.log
set cli timestamp

Nov 30 22:36:28
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0> it's all yours now!:)
show system uptime
Nov 30 22:36:28
Current time: 2014-11-30 22:36:28 EST
System booted: 2014-08-24 17:55:36 EDT (14w0d 05:40 ago)
Protocols started: 2014-11-24 08:58:14 EST (6d 13:38 ago)
Last configured: 2014-11-24 08:51:12 EST (6d 13:45 ago) by root
10:36PM  up 98 days,  5:41, 1 user, load averages: 0.52, 0.26, 0.11

{master}
lab@alecto-re0>

1.15. persistent mode

if use running crtc with -p (or set persist_mode 1 in config file), the session will become a little bit "persistent" - once got disconnected, the script will be able to detect this situation and try the best to re-login again - your session will become very "sticky" into the router and won’t be kicked off anymore :)

press !p in the session will toggle the persistent mode

Example 5. persistent mode

telnet> quit
Connection closed.

will reconnect in 30s
ping@ubuntu1404:~$ telnet b6tsb17.jtac-west.jnpr.net 7021
Trying 172.22.194.102...
Connected to b6tsb17.jtac-west.jnpr.net.
Escape character is '^]'.

Type the hot key to suspend the connection: <CTRL>Z

alecto-re0 (ttyd0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{backup}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{backup}
lab@alecto-re0> set cli timestamp

Nov 24 08:10:40
CLI timestamp set to: %b %d %T

at this time, there is no easy way to kick off the session - even the router was reloaded, switched over, or your vty was cleared intentionally. as soon as the session got disconnected, the script will detect the change and will just try to login again and again "persistently".

1.16. logs

name of log file can be specified with some special chars, which will then be substituted by certain info, e.g:

set log_filename  "%S-%T.log"

This will make log file looks "seahawks-re0@jtac-2016_0715_2254_47.log"

TODO…​

logs will be recorded in file specified with log_fullname option. if this option is not set, use "NAME-OF-SESSION.log" under folder specified by log_dir option.

Example 6. logs

given config option below:

set log_dir "~/att-lab-logs"
set log_fullname "~/abc.log"

the log file will be ~/abc.log

ping@ubuntu47-3:~$ crtc myrouter
<<<CRTC:myrouter:start to login, please wait ...
<<<CRTC:myrouter:to interupt the login process, press <ESC>!
<<<CRTC:myrouter:to exit script(kill): press <ESC> and !Q
telnet -K  alecto.jtac-east.jnpr.net
ping@ubuntu47-3:~$ telnet -K  alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re1 (ttyp0)

login: labroot
Password:

--- JUNOS 11.4R5-S4.7 built 2015-05-06 18:55:53 UTC
labroot@alecto-re1> set cli screen-width 300
Screen width set to 300

labroot@alecto-re1> set cli timestamp
Feb 29 11:34:11
CLI timestamp set to: %b %d %T

labroot@alecto-re1>
<<<CRTC:myrouter:login succeeded!
log file: ~/abc.txt         #<------
<<<CRTC:myrouter:automation done

labroot@alecto-re1>

if not set log_fullname :

set log_dir "~/att-lab-logs"

the log file will be ~/att-lab-logs/myrouter.log , for router myrouter:

ping@ubuntu47-3:~$ crtc myrouter
<<<CRTC:myrouter:start to login, please wait ...
<<<CRTC:myrouter:to interupt the login process, press <ESC>!
<<<CRTC:myrouter:to exit script(kill): press <ESC> and !Q
telnet -K  alecto.jtac-east.jnpr.net
ping@ubuntu47-3:~$ telnet -K  alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re1 (ttyp0)

login: labroot
Password:

--- JUNOS 11.4R5-S4.7 built 2015-05-06 18:55:53 UTC
labroot@alecto-re1> set cli screen-width 300
Screen width set to 300

labroot@alecto-re1> set cli timestamp
Feb 29 11:36:35
CLI timestamp set to: %b %d %T

labroot@alecto-re1>
<<<CRTC:myrouter:login succeeded!
log file: ~/logs/myrouter.log       #<------
<<<CRTC:myrouter:automation done

labroot@alecto-re1>

Note	the folder configured in option log_dir will be created if not existing yet.

1.17. debug mode

TODO:

1.18. nested crtc

TODO

set login_info(hoho@att) [list              \
    "$"      "ssh junotac@135.157.14.229"    \
    "sword"  "m0nday"                        \
    "\\\$"   "ssh -l $attproduction_account 199.37.160.81" \
    "sword"  "$attproduction_pass"                   \
]

set login_info($session@hoho)       [list        \
    "$"         "crtc hoho@att"              \
    "succeed" "ssh -l $attproduction_account $session" \
    "sword"     "$attproduction_pass"   \
    ">"         "set cli timestamp"             \
    ">"         "set cli screen-width 300"             \
]

1.19. command substitutions

• %H will be substituted to host (router name)

• %T will be substituted to current time.

Example 7. save config file to a file name that has a timestamp and router name embeded

crtc -Hv0qa "set prefix_mark 0" -c "config" -c "save backup%H_%T" myrouter

this will backup junos config file with a name including timestamp and router name.

1.20. set "arbitrary" option: -a flag, and !a command

analogous to Expect’s built-in flag -c

expect -c "set debug 1" crtc myrouter

with crtc this can be done as:

crtc -a "set debug 1" myrouter

any other options can be set with -a.

1.21. built-in multi-party kibitz "on the fly"

login a router with crtc

labroot@seahawks-re0>
Jun 12 17:33:07

press !K to send invitation to another user

labroot@seahawks-re0> !K
    - invite a user to share your current terminal session?[lrq<ENTER>]
    (l)ocal user: press "l" or just hit enter
       - send invitation to a user in local server (where crtc script was started)
    (r)emote user:press "r"
       - spawn a new shell and from which you can login to remote host and then send
       invitation to a user in that host
    (q)uit: press "q"
       - quit and return back to current session

1.21.1. local sharing: sharing a terminal session with users in local server

press l or hit enter will send invitation to local user

who are you going to invite?
user1
will invite user1 ...
kibitz succeeded!

<<<CRTC:resuming session of seahawks-re0@jtac...

labroot@seahawks-re0>
Jun 12 17:33:27

now user1 in local machine got connected

user1@ubuntu47-3:~$
Message from ping@ubuntu47-3 on pts/83 at 16:46 ...
Can we talk? Run: kibitz -23312
EOF
kibitz -23312
Escape sequence is ^]
session seahawks-re0@jtac is being shared between ping user1 now...

Jun 12 17:33:23

labroot@seahawks-re0>
Jun 12 17:33:27

now invite a 2nd local user: user2

labroot@seahawks-re0> !K
    - invite a user to share your current terminal session?[lrq<ENTER>]
    (l)ocal user: press "l" or just hit enter
       - send invitation to a user in local server (where crtc script was started)
    (r)emote user:press "r"
       - spawn a new shell and from which you can login to remote host and then send
       invitation to a user in that host
    (q)uit: press "q"
       - quit and return back to current session

who are you going to invite?
user2
will invite user2 ...
kibitz succeeded!

<<<CRTC:resuming session of seahawks-re0@jtac...

labroot@seahawks-re0>
Jun 12 17:33:45

labroot@seahawks-re0> show system uptime
Jun 12 17:35:11
Current time: 2016-06-12 17:35:11 EDT
System booted: 2016-06-06 05:50:22 EDT (6d 11:44 ago)
Protocols started: 2016-06-06 13:25:34 EDT (6d 04:09 ago)
Last configured: 2016-06-06 13:24:03 EDT (6d 04:11 ago) by labroot
 5:35PM  up 6 days, 11:45, 2 users, load averages: 0.15, 0.07, 0.01

labroot@seahawks-re0>

2nd user joined the session

labroot@seahawks-re0> please hold on while ping is trying to invite more users...
session seahawks-re0@jtac is being shared between ping user1 user2 now...

Jun 12 17:33:44

labroot@seahawks-re0>
Jun 12 17:33:45

labroot@seahawks-re0> show system uptime
Jun 12 17:35:11
Current time: 2016-06-12 17:35:11 EDT
System booted: 2016-06-06 05:50:22 EDT (6d 11:44 ago)
Protocols started: 2016-06-06 13:25:34 EDT (6d 04:09 ago)
Last configured: 2016-06-06 13:24:03 EDT (6d 04:11 ago) by labroot
 5:35PM  up 6 days, 11:45, 2 users, load averages: 0.15, 0.07, 0.01

user2@ubuntu47-3:~$
Message from ping@ubuntu47-3 on pts/84 at 16:46 ...
Can we talk? Run: kibitz -23359
EOF
kibitz -23359
Escape sequence is ^]
session seahawks-re0@jtac is being shared between ping user1 user2 now...

Jun 12 17:33:44

labroot@seahawks-re0>
Jun 12 17:33:45

1.21.2. distributed sharing: sharing session with users in remote servers

type r after !K will can start a new shell

labroot@seahawks-re0> !K
    - invite a user to share your current terminal session?[lrq<ENTER>]
    (l)ocal user: press "l" or just hit enter
       - send invitation to a user in local server (where crtc script was started)
    (r)emote user:press "r"
       - spawn a new shell and from which you can login to remote host and then send
       invitation to a user in that host
    (q)uit: press "q"
       - quit and return back to current session
no free shell available, will spawn one...
spawned new shell[24187]...
press \l to list, \NUMBER to switch into a session

type \l to list currently shell available to use

labroot@seahawks-re0> host lists:
0:exp7/23292 1:exp11/24187

type \1 to switch to the new shell#1

labroot@seahawks-re0> switched to session: 1:exp11/24187
ping@ubuntu47-3:~$

from the new shell, login to any remote server and to press !K again to send sharing invitation in the server.

who to invite on this server?
pings ttyla
will invite pings on tty -tty ttyla ...
pings is not logged in yet, check and try again later!
<<<CRTC:resuming session of seahawks-re0@jtac...

the user in remote server accepted the invitation

pings@svl-jtac-tool02:~$
Message from pings@svl-jtac-tool02.juniper.net on ttysm at 14:16 ...
Can we talk? Run: kibitz -95467
EOF
kibitz -95467       #<------
Escape sequence is ^]
session seahawks-re0@jtac is being shared between [ping user1 user2 pings] now...
to exit, type ctrl+], and then "exit"

Jun 12 17:51:26

labroot@seahawks-re0>
Jun 12 17:51:27

now all 5 users (1xoriginal 2xlocal 1xremote) are sharing the same terminal session to router seahawks.

!b can be pressed to toggle the sharing/not sharing.

1.22. performance fine tune

• turn off set expect_matchany 0

• turn off dynamic interact composed from user config: set enable_user_patterns1

• turn off all features under interact: nofeature

• turn off logging: set log_when 0

1.23. misc usages case

with this added in the config file crtc.conf under home dir:

set cmds1(sonata@jtac) {
    {cprod -A fpc0 -c 'set dc bc "getreg chg rdbgc0"'}
    {cprod -A fpc1 -c 'set dc bc "getreg chg rdbgc0"'}
}

this command :

crtc -pr3tiUn 10000 -b "start shell" sonata@jtac

will behaves as following:

• monitor the shell commands 10000 times

• with a 1s interval,

• print a local timestamp for each command,

• before executing cmds provided in the cmds1 array in config file

• if being disconnected (RE switchover, etc):

  • reconnect in 3s

  • restart the commands sequence from all over (-U) (default is continue from where left over)

this is quite similiar with running a shell script from within the junos shell:

while [ 1 ]
do
  date
  cprod -A fpc0 -c 'set dc bc "getreg chg rdbgc0"'
  cprod -A fpc1 -c 'set dc bc "getreg chg rdbgc0"'
  sleep 1
done

this is usefull for the testings when the connections will be kicked off frequently.

1.24. more other working examples

TODO:

crtc -pc "show system uptime" -n 20 -i 5 -r 5 alecto@jtac

crtc -d3n 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
    -S "\$pps == 0" anewrouter

crtc -E "$" -S "telnet alecto.jtac-east.jnpr.net" -E
    "login: " -S "lab" -E "Password:" -S "lab123"
    -E ">" -S "set cli timestamp" -c "show system uptime" -n 3 -i 5
    anewrouter

crtc -n 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
    -I "1@pps==pps_prev" anewrouter

crtc -n 10 -i 20 -R "2@2601\s+rmt\s+LD" anewrouter

#comment out regex_info and issue_info in config, then:
ping@ubuntu1404:~$ crtc -D100yc "show interfaces ge-1/3/4" -n 100 \
-i2 -R "1@@Physical link is (\w+)@updown" -I '1@updown == "Down"' \
-l pings@juniper.net alecto@jtac

2. usage cases

2.1. device login

2.1.1. login to any of your individual device:

add the login steps into the login_info array like below in your config file crtc.conf:

set login_info(myrouter) { "\\\$" "telnet x.x.x.x" "login: " "YOUR_LOGIN_NAME" "Password:" "YOUR_PASS" }

this can be improved:

• the long line can be broken into several shorter lines with a style similiar to Junos config. ^[3]

• the "Password", can be shortened as "sword", to make it safer to match.

• the weird-looking \\\$ can be replaced as $, which will be explained later.

the improved login process looks like this:

set login_info(myrouter) {
    "\\\$" "telnet x.x.x.x"
    "login: " "YOUR_LOGIN_NAME"
    "sword:" "YOUR_PASS"
}

The step reads as "pattern" "action" pairs:

• "expect", or looking for a "$" sign, as soon as found, send a "telnet x.x.x.x" command to the terminal

• "expect" a prompt "login:", then input login name

• "expect" a prompt "sword" , and input password

if the login_info refers other varibles, use list keyword plus \ as continuation indicator,

set your_login_name "ping"
set your_password "password"
set login_info(myrouter)       [list            \
    "$"        "telnet x.x.x.x"                 \
    "login: "      "$your_login_name"           \
    "Password:"    "$your_password"             \
]

then login your router:

~pings/bin/crtc myrouter

TODO: explain $ vs. \\\$.

a real example is showned below.

Example 8. login to a device

the final config :

set domain_suffix_con jtac-west.jnpr.net
set domain_suffix jtac-east.jnpr.net

#test only
set login_info(myrouter)       [list               \
    "$"        "telnet alecto.$domain_suffix"    \
    "login: "      "lab"                        \
    "Password:"    "lab123"                   \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"          \
]

capture of the login process:

ping@ubuntu1404:~$ crtc myrouter
current log file ~/att-lab-logs/myrouter.log
telnet alecto.jtac-east.jnpr.net
ping@ubuntu1404:~$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re0> it's all yours now!:)
set cli timestamp

Nov 30 16:41:38
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0>
Nov 30 16:41:38

{master}
lab@alecto-re0>

Note

you can use variables in the config and do the varible substitutions [4] . in this case 2 vars are used to represent the domain names: set domain_suffix_con jtac-west.jnpr.net set domain_suffix jtac-east.jnpr.net the varible domain_suffix is then used in the login_info array element, and will be substituted into the actual value jtac-west.jnpr.net "$" "telnet alecto.$domain_suffix" \ after login completed (password accepted and login succeeded), more commands can be "attached" in the login_info array. It is handy to always have a bunch of commonly used commands (like set timestamp, terminal width,etc) sent before starting the work

2.1.2. login to any router under a "jtac" class

all routers under a "class" share the same login steps. the only thing differences are the session name or the host name. this make it possible to just add one "class" of login steps for multiple routers.

config:

set domain_suffix_con jtac-west.jnpr.net
set domain_suffix jtac-east.jnpr.net
set jtaclab_login lab
set jtaclab_pass lab123

#define a new category "jtac" for all jtac devices
set login_info($session@jtac)       [list       \
    "$"        "telnet $host.$domain_suffix"    \
    "login: "      "$jtablab_login"             \
    "Password:"    "$jtaclab_pass"              \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"          \
]

login:

ping@ubuntu1404:~$ crtc tintin@jtac
current log file ~/att-lab-logs/tintin.log
telnet tintin.jtac-east.jnpr.net
ping@ubuntu1404:~$ telnet tintin.jtac-east.jnpr.net
Trying 172.19.161.18...
Connected to tintin.jtac-east.jnpr.net.
Escape character is '^]'.
Please contact ...<snipped>...
tintin-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3R3-S4.7 built 2014-06-30 05:41:54 UTC

************MVPN setup - NYPJAR1 ****************

lab@tintin-re0> set cli screen-width 300
Screen width set to 300

lab@tintin-re0> it's all yours now!:)
set cli timestamp

Nov 30 20:12:06
CLI timestamp set to: %b %d %T

2.1.3. login to console of juniper jtac lab routers

the trick is to load the hostmap data provided by lab administrator, which defines which terminal server/port number to be used when login the console of a specific router. to differentiate the console session with a mgmt session, a fake hostname:ROUTERNAME-ren-con is defined. The tcl regexp command is used to extract these info out of the mapping.

config:

#juniper jtac router console login:
#map a (fake) session name to terminal server data
array set hostmap {\
    ......
    eros-re0-con                        b6tsb17:7015    \
    eros-re1-con                        b6tsb17:7016    \
    alecto-re0-con                      b6tsb17:7021    \
    alecto-re1-con                      b6tsb17:7022    \
    rams-re0-con                        b6tsa26:7023    \
    bills-re0-con                       b6tsa26:7024    \
    bears-re0-con                       b6tsb09:7013    \
    chargers-re0-con                    b6tsb09:7014    \
    ......
}

#domains
set domain_suffix_con jtac-west.jnpr.net
set domain_suffix jtac-east.jnpr.net
set jtaclab_login lab
set jtaclab_pass lab123

#analyze the session name and compose the login command:
#for console session: extract "terminal server" name and "port number"
#for mgmt session: use the normal "host" to login
if [regexp {(\w+):(\d+)} $host -> terminal_server port] {
    set login_string "telnet $terminal_server.$domain_suffix_con $port"
} else {
    set login_string "telnet $host.$domain_suffix"
}

#define a new category "jtac" for all jtac devices
set login_info($session@jtac)       [list           \
    "$"        "$login_string"                      \
    "login: "      "$jtablab_login"             \
    "Password:"    "$jtaclab_pass"              \
    ">"            "set cli screen-width 300"       \
    ">"            "set cli timestamp"              \
]

Example 9. login:

ping@ubuntu1404:~$ crtc alecto-re0-con@jtac
current log file ~/att-lab-logs/b6tsb17:7021.log
telnet b6tsb17.jtac-west.jnpr.net 7021
ping@ubuntu1404:~$ telnet b6tsb17.jtac-west.jnpr.net 7021
Trying 172.22.194.102...
Connected to b6tsb17.jtac-west.jnpr.net.
Escape character is '^]'.

Type the hot key to suspend the connection: <CTRL>Z
alecto-re0 (ttyd0)
login: lab
Password:
--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{backup}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{backup}
lab@alecto-re0> set cli timestamp

Nov 24 08:09:27
CLI timestamp set to: %b %d %T

{backup}
lab@alecto-re0> it's all yours now!:)

Nov 24 08:09:28

{backup}
lab@alecto-re0>

Note	as expected, to logout, exit won’t kick yourself out, press the "escape character" ctrl-] to get into telnet client application, and then type quit to exit telnet client {backup} lab@alecto-re0> exit Nov 24 08:10:05 alecto-re0 (ttyd0) login: telnet> quit Connection closed.

2.1.4. login to a customer’s device

usually for security reason, there won’t be direct access to customer’s device. sometime one or two intermediate springboards are used, or a "menu" pop out so you have to select your target device, or sometime a "token" number (with a "PIN") is required to input before proceeding.

here is an example of handling the "menu":

config:

in my own PC:

set login_info(qfx11@att)       [list           \
    "$"        "ssh svl-jtac-tool02"            \
    "$"         "ssh jtac@12.40.233.51"         \
    "assword:"  "PASSWORD"                      \
    "Enter Option:"     "11"                    \
    "assword:"  "PASSWORD"                      \
    ">"         "set cli screen-width 300"      \
    ">"         "set cli timestamp"             \
]

this is to login jtac server first, then login customer’s springboard, from there a menu is popped up for selection. Or, if you run crtc from the svl server already, then no need the 1st line:

set login_info(qfx11@att)       [list           \
    "$"         "ssh jtac@12.40.233.51"         \
    "assword:"  "PASSWORD"                      \
    "Enter Option:"     "11"                    \
    "assword:"  "PASSWORD"                      \
    ">"         "set cli screen-width 300"      \
    ">"         "set cli timestamp"             \
]

Example 10. login:

ping@ubuntu1404:~/bin$ crtc qfx11@att
current log file ~/att-lab-logs/qfx11.log
ssh svl-jtac-tool02
ssh jtac@12.40.233.51
ping@ubuntu1404:~/bin$ ssh svl-jtac-tool02
Last login: Fri Nov 21 10:20:35 2014 from 172.25.163.165
Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
        The Regents of the University of California.  All rights reserved.

FreeBSD 7.4-RELEASE (PAE) #0: Tue Jul 23 08:43:51 PDT 2013

******************************************************************************
**                                                                          **
**      Hostname:               svl-jtac-tool02.juniper.net                 **
**      IP Address:             172.17.31.81                                **
**      Internet Address:       66.129.239.20                               **
**      Operating System:       FreeBSD 7.4-RELEASE                         **
**                                                                          **
******************************************************************************
For operational availability issues, contact helpdesk.
For functional enhancement requests, contact jboyle
******************************************************************************
JTAC Shell Servers:  svl-jtac-tool01, svl-jtac-tool02, wfd-jtac-tool01
******************************************************************************

Saturday Nov 22 12:00p,
  Proactive reboot planned to clean up stray processes after filer issues
ssh jtac@12.40.233.51
[pings@svl-jtac-tool02 ~]$ ssh jtac@12.40.233.51
Warning: Permanently added '12.40.233.51' (DSA) to the list of known hosts.
jtac@12.40.233.51's password:
Last login: Fri Nov 21 12:27:08 2014 from 66.129.239.20
#               For Authorized Use Only
#
# All of activities on this system are monitored. Anyone using this system
# expressly consents to such monitoring and is advised
# that if such monitoring reveals possible evidence of criminal activity,
# system personnel may provide the evidence of such monitoring to law
# enforcement officals and may be executed to the fullest extent of law
#
#
             0.  EXIT
1.  ymappr01            2.  ymappr02
3.  ybpnyubfg           4.  ymafha02
5.  ymappr03            6.  ymappr04
7.  ymappr05            8.  ymappr10
9.  zgcawgnk100         10. zgnawgnk100
11. zggawgnk100         12. atrpgf01
13. atrzsp3             14. atbzsp3
15. atrzsp5             16. atbzsp2
17. ymappr06            18. ymappr07
19. ymappr08            20. atbzsn1
21. atbzsn2             22. atrzsp11
23. zgvawyfvp03           24. pvcppr1
25. atrppr2             26. pvcppr2
27. atrzsp4               28. atbzsn3
29. atbzsn4               30. atBcnp4
31. atrzsp9             32. zgvaw003gf
33. atrzsp10              34. EbhgrFreire
35. atrppr3             36. atrzsp6
38. atrzsp7             38. atrzsp8
39. zgvaw00001ppr9        40. zgvaw00002ppr9
41. zgraw00001ppr9        42.zgraw00002ppr9
43. zgcaw00001ppr9        44.zgcaw00002ppr9
45. zgnaw00001ppr9
Enter Option: 11
                      Warning Notice

Password:
--- JUNOS 13.2X51-D30_vjunos.50 built 2014-11-21 03:45:16 UTC
{master:0}
jtac@zggawgnk100> set cli screen-width 300
Screen width set to 300

{master:0}
jtac@zggawgnk100> set cli timestamp
Nov 22 02:18:05
CLI timestamp set to: %b %d %T

{master:0}
jtac@zggawgnk100>

the password and all other sensitive text info has been encrypted in this example.

2.2. send commands

sometime you prefer always to send some quick commands after logged into a router successfully. this can be done in many different ways:

• just extend the login_info array and attach more commands to be sent[X1]

• use a seperate cmds1 array to hold all extra commands

• use command line option -c

• use command line options: -e and -s

2.2.1. -c option

the -c (command) option is a easy way to send commands right after a successful login, it does not require any config change

config

NONE

this is equivalent to having this settings in config file:

set cmds1(alecto)         [list \
    "show version"              \
    "show system uptime"        \
]

Example 11. login

ping@ubuntu1404:~$ crtc -c "show version" -c "show system uptime" alecto
current log file ~/att-lab-logs/alecto.log
telnet alecto.jtac-east.jnpr.net
ping@ubuntu1404:~$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re0> set cli timestamp

Nov 30 22:06:22
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0> show version
Nov 30 22:06:22
Hostname: alecto-re0
Model: m320
JUNOS Base OS boot [12.3-20140210_dev_x_123_att.0]
...<snipped>...
JUNOS Routing Software Suite [12.3-20140210_dev_x_123_att.0]

{master}
lab@alecto-re0> show system uptime
Nov 30 22:06:22
Current time: 2014-11-30 22:06:22 EST
System booted: 2014-08-24 17:55:36 EDT (14w0d 05:10 ago)
Protocols started: 2014-11-24 08:58:14 EST (6d 13:08 ago)
Last configured: 2014-11-24 08:51:12 EST (6d 13:15 ago) by root
10:06PM  up 98 days,  5:11, 1 user, load averages: 0.00, 0.04, 0.01

{master}
lab@alecto-re0> it's all yours now!:)

Nov 30 22:06:22

Note	this option -c (and also the similiar -e/-s option "two-tuples" can be used multiple times. this is convenient in case more commands need to be sent to the router from the command line.

here is an example of sending longer list of commands

shutdown and wait 3 seconds, then bring up (rollback) after 3 seconds:

~pings/bin/crtc -c "configure" -c "set interfaces xe-3/1/0 disable" -c "commit" -c "SLEEP 3" -c "rollback 1" -c "show | compare" -c "commit" -c "exit" alecto@jtac

this maybe look too long, So the same thing can be done with following configured in ~/crtc.conf:

set pre_cmds1(alecto@jtac) [list \
     "configure"        \
]

set cmds1(alecto@jtac)         [list   \
    "set interfaces xe-3/1/0 disable"   \
    "commit"   \
    "SLEEP 3"   \
    "rollback 1"   \
    "show | compare"   \
    "commit"   \
]

set post_cmds1(alecto@jtac) [list \
     "exit"        \
]

now just run short command without options:

~pings/bin/crtc alecto@jtac

2.2.2. -e and -s options

in addition to configuration file and -c option, the 3rd method is to use -e (expect) and -s (send) option - the script is expecting for a specific prompt (> in this case of Juniper router privilidge mode) specified by -e option, and once that is seen , a command (specified by -s) will then be sent to the router.

A good usage example is that if you need to type in another (root) password to promote yourself with higher privilidge in order to perform some sensitive commands:

vmx:

ping@ubuntu1404:~/bin$ crtc -e "\\\$" -s "su" -e "Password:" -s "lab123" -e "#" -s "ethtool em2" -e "#" -s "exit" vmx
current log file ~/logs/10.85.4.17.log
telnet -K  10.85.4.17
ping@ubuntu1404:~/bin$ telnet -K  10.85.4.17
Trying 10.85.4.17...
Connected to 10.85.4.17.
Escape character is '^]'.
Ubuntu 14.04.1 LTS
MX86-host-BL660C-B1 login: labroot
Password:
Last login: Thu Dec  4 23:09:54 PST 2014 from ubuntu1404.jnpr.net on pts/2
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic x86_64)
labroot@MX86-host-BL660C-B1:~$ date
Thu Dec  4 23:11:13 PST 2014
labroot@MX86-host-BL660C-B1:~$ su
Password:
root@MX86-host-BL660C-B1:/home/labroot# ethtool em2
Settings for em2:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseT/Full
                                10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 10000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
root@MX86-host-BL660C-B1:/home/labroot# exit
exit
labroot@MX86-host-BL660C-B1:~$

or e320:

crtc -e "#" -s "support" -e "Password:" -s "plugh" -e "#" -s "shell" -e "->" -s "memShow" -e "->" -s "exit" -e "#" -s "exit" e320-svl
ping@ubuntu1404:~/bin$ telnet -K  172.19.165.56
Trying 172.19.165.56...
Connected to 172.19.165.56.
Escape character is '^]'.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!   YOU HAVE CONNECTED TO VERIZON, INC.    !
!                                          !
!  UNAUTHORIZED ACCESS WILL BE PROSECUTED  !
!   TO THE FULLEST EXTENT OF THE LAW ! !   !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Telnet password: *******
Logged in on vty 0 via telnet.
Copyright (c) 1999-2014 Juniper Networks, Inc.  All rights reserved.

           !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
           ! NO CHANGES TO THIS CONFIG ARE TO BE MADE !
           !  WITHOUT ENGINEERING OR IP NOC SUPPORT!  !
           !                                          !
           !      TACACS LOGS WILL BE AUDITED! ! !    !
           !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
E320-SVL:vol>enable 15
E320-SVL:vol#
E320-SVL:vol#support
Password: *****
E320-SVL:vol(support)#shell
->
-> it's all yours now!:)

E320-SVL:vol(support)#shell
-> memShow
 status      bytes    blocks   avg block  max block  avg searches
 ------   ---------- -------- ---------- ----------  ------------
current
    free  2928251816     5200     563125 2868448424
    alloc  520366168   251384       2070      -
cumulative
    alloc 3072303840  476687427          6      -           1

value = 0 = 0x0
-> exit
E320-SVL:vol(support)#exit
E320-SVL:vol#

Note

the -c command presumes a "commonly used" prompt value (one of the %>#$ characters) will pop out, which is by default use the following pattern: set pattern_common_prompt {(% |> |# |\$ |%|>|#|\$)$} it seems to be good enough for most of the network devices in this industry. But it can be changed if it doesn’t match the prompt that you are trying to login.

todo: add -v flag for "pattern_common_prompt" option

2.3. automate shell tasks

In this example we use crtc to create a file sync service between two servers. For those who got used to crontab, this is quite close in term of the service provided. But there are extra benefits with crtc:

• much easier to monitor/troubleshoot the scheduled tasks

• able to schedule tasks involving passwords

to sync two pairs of folders between one of jtac server and scooby server, via rsync:

pings@svl-jtac-tool01:~$ crtc -Mj1n10000i30 rsync

it keep running rsync to "post" all files found in an outgoing folder of one end of rsync, to the incoming folder of the other end.

jtac-tool:outgoing   -> scooby:incoming
scooby:outgoing      -> jtac-tool:incoming

configurations

the script will be running in a loop, posting/pulling data between the 2 servers, until it reaches 10000 iterations. The gap between each iteration are 30 seconds. The script is configurable in many ways:

1. via a config file

2. via shell environment variables

3. on the fly when it is running

config file ~pings/bin/crtc.conf

this is the rsync related configuration in the config file crtc.conf:

#between jtac-tools and scooby server
set rsync_log "/volume/CSdata/pings/rsync4jtac.log"
set rsync_outgoing_dir_local "/volume/CSdata/pings/outgoing/"
#set rsync_incoming_dir_remote "/mnt/NAS/media/incoming/"
set rsync_incoming_dir_remote "/incoming/"
set rsync_incoming_dir_local "/volume/CSdata/pings/incoming/"
#set rsync_outgoing_dir_remote "/mnt/NAS/media/outgoing/"
set rsync_outgoing_dir_remote "/outgoing/"
set rsync_remote "12.3.167.13"
set rsync_remote_account "jtac"
set rsync_remote_pass "jnpr"

#-L: always copy file/folder resolved by link
#-P: Partial: continue from where left off
#-c: --checksum, skip based on checksum, not mod-time & size
set rsync_opt "-avvPzLc --log-file=$rsync_log"
set rsync_post "rsync $rsync_opt $rsync_outgoing_dir_local $rsync_remote_account@$rsync_remote:$rsync_incoming_dir_remote"
set rsync_pull "rsync $rsync_opt $rsync_remote_account@$rsync_remote:$rsync_outgoing_dir_remote $rsync_incoming_dir_local"

set cmds1(rsync-scooby) [list                  \
 "\\\$"     "uptime"                 \
 "\\\$"  "$rsync_post"               \
 "sword" "$rsync_remote_pass"        \
 "\\\$"  "$rsync_pull"               \
 "sword" "$rsync_remote_pass"        \
]

changing these configs will change the script running behavior. all changes in the config file will be read by the script and take effect on the fly.

to access the script

The script is running inside a screen session so it can keep running in the server without a client attached (e.g.: telnet/ssh client).

To acquire the access to the script, login to same jtac-tool server (e.g. svl-jtac-tool01 in this case) where the script is running and acquire the view of the screen session:

[nzhao@svl-jtac-tool01 ~]$ export TERM=xterm
[nzhao@svl-jtac-tool01 ~]$ screen -x pings/rsync

This will get the sharing view of the screen terminal where the script is running. all operations under this screen will be seen by any other authorized users .

To obtain the control of the script exclusively:

[nzhao@svl-jtac-tool01 ~]$ export TERM=xterm
[nzhao@svl-jtac-tool01 ~]$ screen -dR pings/rsync

Note	user authorization can be added/removed as needed per request.

shell environment variables

set and export variables from current shell and new value will take effect when crtc runs next time. the name of the shell variable is what was defined in the crtc.conf file, with a prefix CRTC_.

This will change the folders of both end of rsync to new values when crtc is running next time.

pings@svl-jtac-tool01:~$ export CRTC_rsync_outgoing_dir_local="folder1"
pings@svl-jtac-tool01:~$ export CRTC_rsync_incoming_dir_remote="folder2"
pings@svl-jtac-tool01:~$ export CRTC_rsync_incoming_dir_local="folder3"
pings@svl-jtac-tool01:~$ export CRTC_rsync_outgoing_dir_remote="folder4"

pings@svl-jtac-tool01:~$ crtc -Mj1n10000i30 rsync

change the script running status on the fly

by default the script will pause and wait until 30s expires before the next iteration of rsync. to change this behavior:

• press + to increase it by 2s

• press - to decrease it by 2s

• press <ENTER> or any other key to skip the sleep and start next rsync iteration right away.

• press q to pause the iteration

• after iteration paused,

  • press !R to resume the iteration from wherever left

  • press !s to stop the automation

  • press !r to start it all over again

TODO

2.4. file sync between juniper server and ATT server

to ease the file copy/image delivery work between Juniper shell servers and ATT servers, a small script is running to periodically synchronize files between these servers.

                      ATT
                    ..........................................
                    .                       135.16.32.251    .
jtac server ----//--.-- scooby ------------------------ atlas.
              (12.3.167.13) (192.168.46.146)        10.74.18.229
                    .                \              /        .
                    .                 \            /         .
                    .                  \          /          .
                    .                   ATT routers          .
                    .            ATT servers/VMXhypervisors  .
                    .                                        .
                    ..........................................

2.4.1. where is the script and how to run

where

pings@svl-jtac-tool01:~$ ~pings/bin/jtacsync.sh

how to run

pings@svl-jtac-tool01:~$ export PATH=$PATH:~pings/bin/
pings@svl-jtac-tool01:~$ jtacsync.sh

Note	to run jtacsync.sh script(file sync) or crtc(device login) script, export PATH=$PATH:~pings/bin/ has to be executed to include my folder into your PATH.

2.4.2. folders being sync.ed

with the script running the file sync will be started every 10m, on these folders/servers:

shellserver(Juniper): outgoing -> Scooby(att): incoming -> atlas(att): incoming
shellserver(Juniper): incoming <- Scooby(att): outgoing <- atlas(att): outgoing

locations of these folders:

Jtac servers (Juniper shell server):

~pings/incoming         (linking to /volume/CSdata/pings/incoming)
~pings/outgoing         (linking to /volume/CSdata/pings/outgoing)

Scooby (ATT ssh server):

/home/jtac/incoming     (linking to /mnt/sdb4/incoming/)
/home/jtac/outgoing     (linking to /mnt/sdb4/outgoing/)

Atlas (ATT ftp server):

/junos/images/incoming
/junos/images/outgoing

2.4.3. steps to copy a file from juniper server to att device

1. generate a symbolic link @ jtac server “outgoing” folder, pointing to the location of the image that you want to transfer:

   pings@svl-jtac-tool01:~$ cd ~pings/outgoing/
   pings@svl-jtac-tool01:~$ ln –s /…/…/jinstall1.tgz  jinstall1.tgz

2. check @ Scooby server or atlas server’s “incoming” folders (monitor any new files and the size)

   pings@svl-jtac-tool01:~$ crtc scooby@attlab
   jtac@scooby:~$ ls -l incoming/

   pings@svl-jtac-tool01:~$ crtc –H atlas@attlab
   natest@135.16.32.251:/junos/images> ls -lct /junos/images/incoming/

   once new files showing up and sizes are correct, you can login att routers/hypervisor and download files from any of the two servers:

   • atlas (10.74.18.229)

   • Scooby (192.168.46.146)

     1. login att routers:

        pings@svl-jtac-tool01:~$ crtc ROUTERNAME@attlab
        start shell

        //downloading a jinstall from scooby (192.168.46.146)
        scp jtac@192.168.46.146:incoming/jinstall1.tgz ./
        password is "jnpr"

     2. login att VMX host/Hypervisor:

        pings@svl-jtac-tool01:~$ crtc DT401JVPE-host@attlab.hv
        ftp 10.74.18.229
        input username "natest"
        input password "crash"

        //downloading a jinstall from atlas(10.74.18.229)
        ftp> cd /junos/images/incoming
        250 Directory successfully changed.
        ftp> get jinstall1.tgz

2.4.4. steps to "upload" a file (coredump, logs, etc) from att device to juniper shell server

1. login att routers:

   pings@svl-jtac-tool01:~$ crtc ROUTERNAME@attlab
   start shell

   //uploading a coredump tarball to scooby (192.168.46.146)
   scp coredump.tgz jtac@192.168.46.146:outgoing/
   password is "jnpr"

2. login att VMX host/Hypervisor:

   pings@svl-jtac-tool01:~$ crtc DT401JVPE-host@attlab.hv

   //uploading a coredump tarball to atlas (10.74.18.229)
   ftp 10.74.18.229
   input username "natest"
   input password "crash"

   ftp> cd /junos/images/outgoing
   250 Directory successfully changed.
   ftp> put coredump.tgz

the uploaded file will be available in juniper shell server’s incoming folder in a while.

pings@svl-jtac-tool01:~/incoming$ ls
coredump.tgz

2.4.5. the jtacsync.sh script

#!/bin/sh
count=1
while true; do
    echo "running crtc $count times..."
    crtc -Mj1W60n10i30q rsync-scooby
    echo "sleeping 10m..."
    sleep 600
    count=$((count + 1))
done

2.5. scan all routers

configure a new group "jtac.fqdn" to login routers under charge of a certain group. in this test we are trying to pull data from jtac department:

set login_string "telnet $host"
set jtaclab_login labroot
set jtaclab_pass lab123

set login_info($session@jtac.fqdn)       [list       \
    "$"         "$go_jtac_server"           \
    "sword"     $unixpass                       \
    "$"        "$login_string"    \
    "login: "      "$jtaclab_login"             \
    "Password:"    "$jtaclab_pass"              \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"          \
]

put this in a shell script scanjtac.sh:

for i in rockets-re0.ultralab.juniper.net willi-re0.ultralab.juniper.net\
    flip.ultralab.juniper.net redskins-re0.ultralab.juniper.net; do

crtc -HW5w5qa \
   -c "show version" \
   -c "show chassis hardware | no-more" \
$i@jtac.fqdn;

done;

run the script:

ping@ubuntu47-3:~$ jtacscan.sh > scan-0325.txt

<<<CRTC:mango-re0.ultralab.juniper.net@jtac.fqdn:start to login, please wait ...
<<<CRTC:mango-re0.ultralab.juniper.net@jtac.fqdn:to interupt the login process, press <ESC>!
<<<CRTC:mango-re0.ultralab.juniper.net@jtac.fqdn:to exit script(kill): press <ESC> and !Q

<<<CRTC:mango-re0.ultralab.juniper.net@jtac.fqdn:login succeeded!
log file: ~/logs/mango-re0.ultralab.juniper.net.log
<<<CRTC:mango-re0.ultralab.juniper.net@jtac.fqdn:automation done
bye!:)

<<<CRTC:bombay-re1.ultralab.juniper.net@jtac.fqdn:start to login, please wait ...
<<<CRTC:bombay-re1.ultralab.juniper.net@jtac.fqdn:to interupt the login process, press <ESC>!
<<<CRTC:bombay-re1.ultralab.juniper.net@jtac.fqdn:to exit script(kill): press <ESC> and !Q

<<<CRTC:bombay-re1.ultralab.juniper.net@jtac.fqdn:login succeeded!
log file: ~/logs/bombay-re1.ultralab.juniper.net.log
<<<CRTC:bombay-re1.ultralab.juniper.net@jtac.fqdn:automation done
bye!:)
...<snippet>...

one thing that can be noticed is that the redirection > seems "not work" in here, because we are still seeing the messages from crtc even after we direct the output to a file. To verify this we can "tail" the file scan-0325.txt to see if the data had been redirected into the file correctly.

ping@ubuntu47-3:~$ tail -f scan-0325.txt
show version
Mar 26 02:13:45
Hostname: mango-re0
Model: mx960
JUNOS Base OS boot [12.1X43.15]
......

{backup}
labroot@mango-re0>

show version
Mar 26 02:53:07
Hostname: bombay-re1
Model: mx960
JUNOS Base OS boot [12.1X43.15]
......

so actually everything works. What happened is CRTC selectively send some messages to /dev/tty instead of standard output . The benefit is that these messages will be processed seperatedly and won’t be "merged" with the data pulled from the devices when the script was redirected to a file - this provides a good indication about the whole progress of the script.

Tip	This seperation can be turned off by -A knob.

2.6. problem replication

add following configurations in the config file: crtc.conf

set easyvar 1
set max_rounds 10
set interval_cmds 20

#login steps
set login_info(anewrouter)       [list          \
    "$"        "telnet alecto.$domain_suffix"   \
    "login: "      "lab"                        \
    "Password:"    "lab123"                   \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"          \
]

#commands to be executed after login
set cmds1(anewrouter) {
    "show interfaces ge-1/3/0.601 extensive | match pps"
    "show vpls connections instance 13979:333601"
}

#define what info to be captured in the cmd output, via regex

#for the 1st cmd(show interface), capture packets and pps counters, save in
#variables named #"packets" and "pps", respectively
set regex_info(1) "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"

#for the 2nd cmd(show vpls), capture a match pattern as "rmt   LD",
#indicating a "remote down" vpls status
set regex_info(2) "2@@2601\s+rmt\s+LD"

#define what should be treated as an "issue": in this example it reads:
#"if var pps value got changed - (not same as its previous value), then we
#hit the issue!"
set issue_info(1) "1@pps!=pps_prev"

#if issue got replicated, send following commands as actions:
set collect(anewrouter) [ list                              \
    "show interfaces ge-1/3/0.601 extensive | no-more"  \
]

#otherwise (issue not appear), send other commands as actions before next
#iteration
set test(anewrouter) [ list                 \
    "configure private"                     \
    "run show system uptime"                \
    "exit"                                  \
]

now run crtc to login the router and start the replication:

crtc anewrouter

what exactly it does are:

1. login to a router named "anewrouter", with the steps configured in login_info array

2. once login succeeds, it then sends commands configured in the cmds1 array

3. repeat this 10 times (max_rounds) , with 20s pause between each iteration (interval_cmds)

4. in each iteration, try to use a regex (regex_info)to examine "issues" in each command output:

   • in the first command, extract traffic counters and put int the provided varible named packets and pps respectively

   • in the 2nd command, use another pattern "rmt LD" to check vpls states

5. define the issue (issue_info) as any one of ^[5] the following two criterias being met:

   • pps (traffic rate) ever changes

   • VPLS remote connection status matches "rmt LD" pattern (remote down)

6. if the issue being reproduced , the collect array will be executed and following will happen:

   • run "show interfaces …​" command

   • send an email notificaiton to the user

   • script exit (or other actions if configured)

7. otherwise (issue not "seen"), the test array will be executed and following will happen:

   • run "configure" command

   • run "run show system uptime" command

   • run "exit" command

   • repeat above proess

TODO: to add more explanations to following examples…​

-x option change the default rule of defining an issue as "any one of" the defined criterias being satisfied, to "all" defined criterias need to be satisfied (regex being matched).

as long as ge-1/3/4 status became "Up", shut it down

crtc -yn 10 -i 20 \
    -c "show interfaces ge-1/3/4 | match Physical" \
    -R "1@@Physical link is (\w+)@updown" \
    -I {1@updown == "Up"} \
    -Y "configure" -Y "set interfaces ge-1/3/5 disable" -Y "commit" \
    -Y "exit" -Y "show interfaces ge-1/3/5" alecto@jtac\

while backup link remains up (and forwarding traffic), as soon as master link ge-1/3/4 came up, switch AE traffic from backup link ge-1/3/5 to the primary.

crtc -xyn 10000 -i 0 -C NONE \
  -c "show interfaces ge-1/3/4 | match Physical" \
  -c "show interfaces ge-1/3/5 | match Physical" \
  -R "1@@Physical link is (\w+)@updown1" \
  -R "2@@Physical link is (\w+)@updown2" \
  -I '1@updown1 == "Up"'  \
  -I '2@updown2 == "Up"' \
  -Y "configure" \
  -Y "set interfaces ge-1/3/5 disable"  \
  -Y "commit" \
  -Y "exit" \
  -Y "show interfaces ge-1/3/5" \
  alecto@jtac

while link ge-1/3/5 was down, as soon as master link ge-1/3/4 went down, remove the "disable" command under ge-1/3/5 (to bring it up and forward the traffic)

crtc -xyn 10000 -i 0 -C NONE \
  -c "show interfaces ge-1/3/4 | match Physical" \
  -c "show interfaces ge-1/3/5 | match Physical" \
  -R "1@@Physical link is (\w+)@updown1" \
  -R "2@@Physical link is (\w+)@updown2" \
  -I '1@updown1 == "Down"'  \
  -I '2@updown2 == "Down"' \
  -Y "configure" \
  -Y "delete set interfaces ge-1/3/5 disable"  \
  -Y "commit" \
  -Y "exit" \
  -Y "show interfaces ge-1/3/5" \
  alecto@jtac

2.7. full parameterization

this example demonstrate the "full parameterization" capability of the crtc script.

let’s say if you don’t want to bother creating or editing a config file, most of the config options in the config file can also be refered as command line options.

So same exact test listed in the previous example can be performed with this long one-liner command:

crtc -yn 10 -i 20 \
  -E "$" -S "telnet alecto.jtac-east.jnpr.net" -E "login: " -S "lab" \
  -E "Password:" -S "lab123" -E ">" \
  -c "show interfaces ge-1/3/0 extensive | match pps" \
  -c "show vpls connections instance 13979:333601" \
  -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps" \
  -R "2@@2601\s+rmt\s+LD" \
  -I "1@pps!=pps_prev" \
  -Y "show interfaces ge-1/3/0.601 extensive | no-more" \
  -N "configure" -N "run show system uptime" -N "exit" \
  a_new_router

1. login to a router named "anewrouter", with the steps configured by -E and -S options pairs

2. once login succeeds, it then sends two commands provided by -c options,

3. repeat this 10 times, with 20s pause between each iteration (-n 10 -i 20)

4. in each iteration, try to use the regex (-R) to match the command output of each:

   • in first command, check traffic counters:"packets" and "pps"

   • in 2nd command, check vpls states: "rmt LD"

5. declare the issue (-I) being reproduced if any one of the following two criterias being met:

   • pps (traffic rate) ever changes

   • VPLS remote connection status ever became "LD"

6. is the issue reproduced?

7. if yes (-Y) the following will be performed:

   • run "show interfaces …​" command (-Y)

   • send an email notificaiton to the user

   • script exit

8. otherwise (-N), run more following test

   • run "configure" command

   • run "run show system uptime" command

   • run "exit" command

   • repeat above proess

2.7.1. compromise between config file and command line options

with following configured in ~/crtc.conf :

set login_info(anewrouter)       [list          \
    "$"        "telnet alecto.$domain_suffix"   \
    "login: "      "lab"                        \
    "Password:"    "lab123"                   \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"          \
]

set cmds1(anewrouter) {
    "show interfaces ge-1/3/0.601 extensive | match pps"
    "show vpls connections instance 13979:333601"
}

the following command monitor either of the 2 commands, if the pps was found to be 0 in the 1st command, then the issue got reproduced.

crtc -n 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
    -I "\$pps == 0" anewrouter

same as above, but the issue will be thought of as "reproduced" as long as the pps value changes:

crtc -yn 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
    -I "1@pps!=pps_prev" anewrouter

monitor the 2nd command to see if remote vpls connection appears "LD" status

crtc -n 10 -i 20 -R "2@@2601\s+rmt\s+LD" anewrouter

another difference between these 2 commands is that, the 2nd one used "-y" command line option, mapping to easyvar config option. with this option you don’t need to use $ when refering to a varible — this avoids the need to have to escape a $ with \ in UNIX shell. so these 2 are equal:

crtc -....(options without a -y)... -I "1@\$var1...\$var2" anewrouter
crtc -y...(other options)... -I "1@var1...var2" anewrouter

2.8. capture a specific values

This example:

crtc -H3yn 10 -i 20 \
    -E "$" -S "telnet alecto.jtac-east.jnpr.net" \
    -E "login: " -S "lab" -E "Password:" -S "lab123" -E ">" \
    -c "show interfaces ge-1/3/0.601 extensive | match pps" \
    -c "show vpls connections instance 13979:333601" \
    -R "1@1@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps" \
    -I "1@pps==pps_prev" -R "2@@2601\s+rmt\S+LD" \
    -Y "show interfaces ge-1/3/0.601 extensive | no-more" \
    -N "configure" -N "run show system uptime" -N "exit" \
    -V "\$pps \$packets" anewrouter

will do the same test as above examples, except that:

• it won’t display the interactions (-H3 : "hide" even more interactions)

• if the issue does not occur, nothing will be displayed

• if the issue occurs, it will print the monitored "packet" and "pps" value in one line output.

a simpler example:

just capture pps and packets 20 times:

crtc -H3yn 10 -i 20 \
    -E "$" -S "telnet alecto.jtac-east.jnpr.net" \
    -E "login: " -S "lab" -E "Password:" -S "lab123" -E ">" \
    -c "show interfaces ge-1/3/0 extensive | match pps" \
    -R "1@2@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps" \
    -V "\$pps \$packets" anewrouter

2.9. timestamping all commands

config

this is equivalent to having this setting in config file:

set timestamp 1

-t option

ping@ubuntu1404:~/bin$ crtc -tH earth@jtac
current log file ~/logs/earth.log
it's all yours now!:)
set cli timestamp

Dec 02 07:58:32
CLI timestamp set to: %b %d %T

lab@earth-re0>
Dec 02 07:58:32

it looks no much different in the beginning. Now typing a return you will now get 2 timestamps:

lab@earth-re0>
Dec 02 07:57:59 2014(local)         #<------
Dec 02 07:58:40                     #<------

lab@earth-re0>

the 2nd one is the original timestamp provided by Junos set cli timestamp command. the 1st one is provided by crtc script , using the local time wherever the script is running. you might notice the slight time difference between the two ^[6]

lab@earth-re0> show system alarms
Dec 02 08:33:13 2014(local)
Dec 02 08:33:54
2 alarms currently active
Alarm time               Class  Description
2014-12-02 01:05:57 EST  Major  FEB 1 Failure
2014-12-02 00:57:34 EST  Major  FEB not online for FPC 1

now, every command invoked by a carriage return will also invoke this extra local timestamp in the output, and will be recorded in the log file.

having redundent timestamp (in Junos) looks stupid, the extra timestamp provided by crtc can be disabled by typing !t (toggle timestamping).

lab@earth-re0> !t
timestamp 0

Dec 03 08:52:09

lab@earth-re0> show system alarms
Dec 03 08:50:08
2 alarms currently active
Alarm time               Class  Description
2014-12-02 01:05:57 EST  Major  FEB 1 Failure
2014-12-02 00:57:34 EST  Major  FEB not online for FPC 1

lab@earth-re0>

Note

This might look unnecessary since Junos already provided this feature. However, not all network devices support this feature - think about other "non-junos" devices which don’t come with this feature by default - cisco , unix, Junos-e, older version of Junos devices (before 7.x?), or even junos shell mode .., this might be a handy trick to timestamp all of your command for easier analysis later on.

2.9.1. timestamp command from non-Junos devices

Example 12. timestamping Juniper E-serials

slot 16->print__11Ic1Detector
Dec 02 15:25:14 2014(local)         #<------timestamp

state=NORMAL
sysUpTime=0x03B33619
passiveLoader=0x0C001994
crashPusherRequested=1
crashPusherReady=1

                         Detect         Credit   Last       Ask      Last
Detector Name            Count  Credits Every    Credit     Every    Ask
------------------------ ------ ------- -------- ---------- -------- ----------
Hw2CamClassifierDetector      0   5/  5    300 s 0x0000A942  5000 ms 0x03B3345D

!t command

suppose you’ve logged into the remote device, but forgot to invoke crtc with -t option. Now you realize you need this feature. without exiting current session, just press !t (an exclamation mark and ! and a letter t literally), you will have this feature. press !t again will toggle it off.

Example 13. timestamping UNIX server

[pings@svl-jtac-tool02 ~]$ crtc -H vmx
current log file ~/logs/10.85.4.17.log
it's all yours now!:)

date

Last login: Tue Dec  2 08:32:42 PST 2014 from svl-jtac-tool02.juniper.net on pts/5
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic x86_64)

Documentation:  https://help.ubuntu.com/

System information as of Tue Dec  2 08:32:42 PST 2014

System load:    16.34               Users logged in:       1
Usage of /home: 78.4% of 343.02GB   IP address for virbr0: 192.168.122.1
Memory usage:   15%                 IP address for br-ext: 10.85.4.17
Swap usage:     0%                  IP address for br-int: 172.16.0.3
Processes:      618

Graph this data and manage this system at:
  https://landscape.canonical.com/

110 packages can be updated.
60 updates are security updates.

labroot@MX86-host-BL660C-B1:~$ date
Tue Dec  2 08:34:08 PST 2014
labroot@MX86-host-BL660C-B1:~$
labroot@MX86-host-BL660C-B1:~$
labroot@MX86-host-BL660C-B1:~$
labroot@MX86-host-BL660C-B1:~$ uptime
 08:34:17 up 33 days, 21:34,  6 users,  load average: 16.39, 16.35, 16.34
labroot@MX86-host-BL660C-B1:~$
labroot@MX86-host-BL660C-B1:~$
timestamp 16-host-BL660C-B1:~$ !t

Dec 02 15:35:43 2014(local)         #<------timestamp turned on
labroot@MX86-host-BL660C-B1:~$ uptime
Dec 02 15:35:45 2014(local)
 08:34:22 up 33 days, 21:34,  6 users,  load average: 16.36, 16.35, 16.33
labroot@MX86-host-BL660C-B1:~$ uname -a
Dec 02 15:35:52 2014(local)
Linux MX86-host-BL660C-B1 3.13.0-32-generic #57-Ubuntu <...truncated...>

Example 14. timestamp Junos shell/PFE commands

lab@mx86-jtac-lab> set cli screen-width 300
Screen width set to 300

lab@mx86-jtac-lab> it's all yours now!:)
set cli timestamp
Dec 03 08:20:24
CLI timestamp set to: %b %d %T
lab@mx86-jtac-lab> start shell      #<------the last timestamped Junos "CLI command"
Dec 03 08:20:30
% ls -l /var/log/messages           #<------no timestamp
-rw-rw----  1 root  wheel  4194094 Dec  3 08:21 /var/log/messages
%                                   #<------press !t (may not display)
timestamp 1                         #<------now timestamp option is set

Dec 03 05:27:37 2014(local)
% ls -l /var/log/messages
Dec 03 05:27:47 2014(local)         #<------shell/pfe commands now both got timestamped
-rw-rw----  1 root  wheel  4195050 Dec  3 08:25 /var/log/messages
root@mx86-jtac-lab% vty 1
Dec 03 05:28:17 2014(local)

BSD platform (Pentium processor, 1536MB memory, 0KB flash)

VMXE(mx86-jtac-lab vty)# show pfe statistics traffic
Dec 03 05:28:35 2014(local)
PFE Traffic statistics:
                    0 packets input  (0 packets/sec)
                    0 packets output (0 packets/sec)

PFE Local Traffic statistics:
...<snipped>...

2.10. features

simplest interact, no extensive pattern match in interact function call. use this to disable all "features" under interactive mode, and get the best performance.

• 2: rich-features (slow/monitoring mode)

  • generate interact code dynamically based on user patterns

  • set enable_user_patterns 3

  • a keystroke will move to fast mode

• 1: flexible

  • ignore user pattern, only use static interact code

  • set enable_user_patterns 1

  • timeout will move to slow mode

• 0: no features (fast mode)

no features

fastest, but no features under interact mode

TODO

rich-features

with most features under interact, maybe slow in some system

TODO

about "flexible features":

when crtc init, it goes to monitoring mode; a user keystroke (any keyboard hit) will move it to fast mode; when user stop typing, after some timeout it will move back to slow/monitoring mode again.

2.11. enable_user_patterns

when set, crtc script will read and parse user_patterns array in the config file, and generate dynamic expect and interact code which can detect user configurable patterns and take the corresponding actions configured for these patterns. This can be very powerful and useful in some cases. e.g. you want the script to automatically reconnect whenever a "Connection closed" pattern is "seen", below code configured in crtc config file will suffice in many cases:

set user_patterns(pattern_connection_close_msg) \
    [list     \
    {Connection closed.*\r|Connection to (\d{1,3}\.){3}\d{1,3} closed|Connection to \S+ closed|Connection reset by peer}  \
    RECONNECT interact_only]

in some other cases you want the script to "retry" the same command over and over until succeed, you can use below config:

set user_patterns(connection_timeout)  \
    {"Connection timed out; remote host not responding" RETRY expect_only}

combining these configs will generate an interesting "persistent" effect of the script:

VMXE(DT408JVPE vty)# show jnh 0 pool summary
                Name        Size      Allocated     % Utilization
               EDMEM    33554432        7715791               22%
               IDMEM      317440         289384               91%
                OMEM    33554432          37288             < 1%%
         Shared LMEM    33555456       33591820              100%

VMXE(DT408JVPE vty)#
VMXE(DT408JVPE vty)# show jnh 0 pool usage
[Connection to dt408jvpe closed                 (1)

<<<interact: detected event:                    (2)
<<<-Connection to dt408jvpe closed-
<<<which matches defined pattern(event):
-<<<Connection closed.*\r|Connection to (\d{1,3}\.){3}\d{1,3} closed|Connection to \S+ closed|Connection reset by peer-
action is RECONNECT                             (3)
persistent mode set 3, will reconnect in 10s
type !p to toggle persistent mode               (4)
<<<<count {10}s before proceeding...            (5)
<<<<  "q":break(quit automation) "Q":exit the script " |\r" (or anything else): continue(escape sleeping)
.........[Fri Jun 10 12:28:29 EDT 2016]:[dt408jvpe@attlab]:myinteract:..process - exp7(in) :exp7(out) :kibitz()..
set cli timestamp
Jun 10 12:41:02
CLI timestamp set to: %b %d %T

j-tac-ps1@DT408JVPE> Read from remote host 12.3.167.8: Operation timed out (6)
Connection to 12.3.167.8 closed

<<<interact: detected event:
<<<-Connection to 12.3.167.8 closed-
<<<which matches defined pattern(event):
-<<<Connection closed.*\r|Connection to (\d{1,3}\.){3}\d{1,3} closed|Connection to \S+ closed|Connection reset by peer-
action is RECONNECT                             (7)
persistent mode set 3, will reconnect in 10s
type !p to toggle persistent mode
<<<<count {10}s before proceeding...
<<<<  "q":break(quit automation) "Q":exit the script " |\r" (or anything else): continue(escape sleeping)
.........[Fri Jun 10 23:11:35 EDT 2016]:[dt408jvpe@attlab]:myinteract:..process - exp7(in) :exp7(out) :kibitz()..
set cli timestamp
Jun 10 23:24:10
CLI timestamp set to: %b %d %T

j-tac-ps1@DT408JVPE>
[Connection to dt408jvpe closed                 (8)

<<<interact: detected event:
<<<-Connection to dt408jvpe closed-
<<<which matches defined pattern(event):
-<<<Connection closed.*\r|Connection to (\d{1,3}\.){3}\d{1,3} closed|Connection to \S+ closed|Connection reset by peer-
action is RECONNECT
persistent mode set 3, will reconnect in 10s
type !p to toggle persistent mode
<<<<count {10}s before proceeding...
<<<<  "q":break(quit automation) "Q":exit the script " |\r" (or anything else): continue(escape sleeping)
.....detected -Connection timed out; remote host not responding-        (9)
user_action configured as RETRY                 (10)
will repeat same command and continue...
.press ctrl-c to get a prompt >$...

......

<<<<count {10}s before proceeding...
<<<<  "q":break(quit automation) "Q":exit the script " |\r" (or anything else): continue(escape sleeping)
detected -Connection timed out; remote host not responding-
user_action configured as RETRY                 (11)
will repeat same command and continue...
.press ctrl-c to get a prompt >$...

<<<<count {10}s before proceeding...
<<<<  "q":break(quit automation) "Q":exit the script " |\r" (or anything else): continue(escape sleeping)
set cli timestamp                               (12)
Jun 12 01:09:52
CLI timestamp set to: %b %d %T

j-tac-ps1@DT408JVPE>
Jun 13 06:43:13

1. TODO <2> <3> <4> <5> <6> <7> <8> <9> <10> <11> <12>

powerful as it looks, this feature may severely lower down the performance of the script. the reason is that if there are too many pattern in the config file, both expect and interact function call of the script will have a big amount of patterns attached, and since for each pattern the Expect has to look for a match anytime a new chunk of characters are received, this effectively slow down the script. disable this feature in the case that performance is a concern.

config file:

set enable_user_patterns 0

inline command: !u

2.12. expect_matchany

"expect_matchany" can tell the diff between a stalled flow or a "flowing" flow without a pattern match (usually in the end when the command prompt returns). expect_matchany first uses .+ to collect chars and then look for pattern match.

expect_matchany shortens the big "waittime_cmd", which otherwise needs to be set to relatively long, to handle the "big commands" that take long time to return. with expect_matchany, it can "recognize" something is still ongoing if the command gives info indicating the progress - just any popped out messages will indicate that the command execution is going well but just not finished yet, and it’s not a condition of "getting stuck" in somewhere…​

Tip	test shows it’s acutally not that slow - definitely not loop for every received character, as I presumed before…​

issues:

how to emulate the "timeout" special pattern?

current implementation: don’t look for "timeout" literally from the match within ".+" match. do it outside.

2.13. prefix_mark

prefix_mark can be used to "insert" some certain strings into each line of a command output - making log files "grep friendly". so it is useful when searching info from log files with grep.

2.14. auto re-attemp with diff account

add this in crtc.conf:

set reconnect_eval {

global session
global SKIP_retry1

if ![info exists SKIP_retry1] {
    set attlab_account $attlab_account2
    set attlab_pass $attlab_pass2
    set SKIP_retry1 1
    puts "retry $SKIP_retry1 time!"
} elseif {$SKIP_retry1==1} {
    set attlab_account $attlab_account3
    set attlab_pass $attlab_pass3
    set SKIP_retry1 2
    puts "retry $SKIP_retry1 time!"
} elseif {$SKIP_retry1==2} {
    set attlab_account $attlab_account4
    set attlab_pass $attlab_pass4
    set SKIP_retry1 3
    puts "retry $SKIP_retry1 time!"
} else {
    set attlab_account $attlab_account5
    set attlab_pass $attlab_pass5
    unset SKIP_retry1
    puts "too much wrong login, will exit..."
}

set login_info(myrouter) [list                  \
    "$" "telnet alecto-re0.$domain_ultralab"    \
    "login: " "$jtaclab_account"                \
    "Password:" "$jtaclab_pass"                 \
    ">" "set cli screen-width 300"              \
    ">" "set cli timestamp"                     \
]

set login_info($session@attlab)       [list        \
    "$"         "ssh jnpr-se@12.3.167.8"        \
    "assword"  "$jnprse_pass"                   \
    ">$"           "$host"                      \
    "login: "      "$attlab_account"               \
    "Password:"    "$attlab_pass"                  \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"   \
]

if !$in_jserver {
    set login_info($session@attlab) [subst -nobackslashes {    \
        "$"     "ssh pings@$juniperserver"      \
        "sword" {$unixpass}                     \
        $login_info($session@attlab)            \
    }]
}

}

set user_patterns(login_retry_diff_account)             [list "ogin: $" "RECONNECT $reconnect_eval"]

when login devices under @attlab domain, crtc will try multiple different account when ran into errors.

2.15. share output to other terminal or a file

redirect to other terminal

1. get terminal file name via stty command

2. redirect crtc script to the other terminal file

labroot@alecto-re0> !Quit the script!                       │labroot@alecto-re0>
ping@ubuntu47-3:~$                                          │May 07 13:53:38
ping@ubuntu47-3:~$                                          │
ping@ubuntu47-3:~$                                          │labroot@alecto-re0> show system uptime
ping@ubuntu47-3:~$ crtc myrouter > /dev/pts/90              │May 07 13:55:01
<<<CRTC:myrouter:start to login, please wait ...            │Current time: 2016-05-07 13:55:01 EDT
<<<CRTC:myrouter:to interupt the login process, press <ESC>!│Time Source:  LOCAL CLOCK
<<<CRTC:myrouter:to exit script(kill): press <ESC> and !Q   │System booted: 2016-04-21 20:47:46 EDT (2w1d 1
                                                            │Protocols started: 2016-04-21 20:49:32 EDT (2w
<<<CRTC:myrouter:login succeeded!                           │Last configured: 2016-04-21 20:56:06 EDT (2w1d
log file: ~/logs/myrouter.log                               │ 1:55PM  up 15 days, 17:07, 2 users, load aver
<<<CRTC:myrouter:automation done                            │
                                                            │labroot@alecto-re0>

redirect crtc to a normal file

share the interactio to multiple people in real time:

ping@ubuntu47-3:~$ crtc -a "set double_echo 1" myrouter > temp-double-echo.txt
<<<CRTC:myrouter:start to login, please wait ...
<<<CRTC:myrouter:to interupt the login process, press <ESC>!
<<<CRTC:myrouter:to exit script(kill): press <ESC> and !Q

<<<CRTC:myrouter:login succeeded!
log file: ~/logs/myrouter.log
<<<CRTC:myrouter:automation done
set cli timestamp
May 07 14:43:06                                     │ping@ubuntu47-3:~$ tail -f temp-double-echo.txt
CLI timestamp set to: %b %d %T                      │labroot@alecto-re0>
                                                    │May 07 14:44:11
labroot@alecto-re0> show version                    │labroot@alecto-re0> show version
May 07 14:43:08                                     |May 07 14:43:08
Hostname: alecto-re0                                |Hostname: alecto-re0
Model: m320                                         |Model: m320
Junos: 15.1R3.6                                     |Junos: 15.1R3.6
JUNOS Base OS boot [15.1R3.6]                       |JUNOS Base OS boot [15.1R3.6]
JUNOS Base OS Software Suite [15.1R3.6]             |JUNOS Base OS Software Suite [15.1R3.6]
JUNOS platform Software Suite [15.1R3.6]            |JUNOS platform Software Suite [15.1R3.6]
JUNOS Web Management [15.1R3.6]                     |JUNOS Web Management [15.1R3.6]

2.16. semi-automation

configure this in config file crtc.conf:

set login_info(myrouter) [list                  \
    "$" "telnet alecto-re0.$domain_ultralab"      \
    "login: " "USER_INPUT"                  \
    "Password:" "USER_INPUT"                 \
    ">" "set cli screen-width 300"              \
    ">" "set cli timestamp"                     \
]

and run crtc:

ping@ubuntu47-3:~$ crtc myrouter
<<<CRTC:myrouter:start to login, please wait ...
<<<CRTC:myrouter:to interupt the login process, press <ESC>!
<<<CRTC:myrouter:to exit script(kill): press <ESC> and !Q
ssh -o "StrictHostKeyChecking no" pings@172.17.31.80
ping@ubuntu47-3:~$ ssh -o "StrictHostKeyChecking no" pings@172.17.31.80
Password:
please input answer to "login: ":   (1)

Warning: No xauth data; using fake authentication data for X11 forwarding.
telnet -K alecto-re0.ultralab.juniper.net
...<snipped>...
   from JTAC labs, more info: http://quickstart.jtaclabs.juniper.net/

pings@svl-jtac-tool01:~$ telnet -K alecto-re0.ultralab.juniper.net
Trying 172.19.161.101...
Connected to jtac-m320-r005.ultralab.juniper.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login: wronglogin
wronglogin
Password:
please input answer to "Password:":
wrongpass

Login incorrect
login: labroot
labroot
Password:lab123


--- JUNOS 15.1R3.6 built 2016-03-24 18:39:40 UTC
labroot@alecto-re0> set cli screen-width 300
Screen width set to 300

labroot@alecto-re0>
<<<CRTC:myrouter:login succeeded!

log file: ~/logs/myrouter.log
<<<CRTC:myrouter:automation done
set cli timestamp
May 09 00:57:40
CLI timestamp set to: %b %d %T

labroot@alecto-re0>

2.17. event script

Configuring Event script:

1. Add this in ~pings/bin/crtc.conf file:

router attga301me6@attlab

#These defined events to be monitored:
#PIM log:
set event_pim {rpd\[\d+\]: %DAEMON-5-RPD_PIM_NBRDOWN: Instance PIM\S+ PIM neighbor[^&]+removed due to:[^&]+ hold-time period}
#BGP log:
set event_bgp {rpd\[\d+\]: %DAEMON-4: bgp_hold_timeout:\d+: NOTIFICATION sent to [^&]+Hold Timer Expired Error}

#These defined actions(cmds) to be executed when any of the events is “seen”:

set ukern_trace(attga301me6@attlab)            {
    "start shell pfe network fpc0"
    "sh ukern_trace 45"
    "sh ukern_trace 46"
    "sh ukern_trace 47"
    "sh ukern_trace 65"
    "exit"
}

#These “bind” each event to its associated action:
set eventscript(attga301me6@attlab)       [list      \
    "$event_pim"        "ukern_trace"      \
    "$event_bgp"        "ukern_trace"      \
]

Same for router chgil302me6@attlab …

#######router chgil302me6@attlab ###############

set ukern_trace(chgil302me6@attlab)            {
    "start shell pfe network fpc0"
    "sh ukern_trace 45"
    "sh ukern_trace 46"
    "sh ukern_trace 47"
    "sh ukern_trace 65"
    "exit"
}

set eventscript(chgil302me6@attlab)       [list      \
    "$event_pim"        "ukern_trace"      \
    "$event_bgp"        "ukern_trace"      \
]

more routers can be added like this

1. Now Run crtc script to monitor the two routers:

   #terminal1:
   pings@svl-jtac-tool01:~$ ~pings/bin/crtc attga301me6@attlab

   #terminal2:
   pings@svl-jtac-tool01:~$ ~pings/bin/crtc achgil302me6@attlab

each of the script now will monitor the defined event (log), and if any thing showing in your terminal matches to the event, the actions (cmds) will be executed

1. Type “monitor start message” in each router, to display the logs in real time, so crtc script can “monitor”.

test:

you can do “file show test” in attga301me6@attlab, and you will see the action will get triggered…

Logs:

Everything you see from your terminal, will be recorded in ~logs/ROUTERNAME.log (not ROUTERNAME@attlab). e.g.: ~/logs/attga301me6.log

2.18. monitor device

pings@svl-jtac-tool01:~$ ~pings/bin/crtc -Aj2n1000i600 attga302me6@attlab

Whait does:

1. Login att router attga302me6

2. send cmds configured in "project 2” (-j2)

3. for any command, print complete output despite of the paging pause (-A)

4. send them 1000 iterations, with internals of 10min (-n1000i600)

cmds configured in “project 2” is in ~pings/bin/crtc.conf:

#monitoring att router for hongpeng {{{2}}}
set vmx_pfe_cli(attga302me6@attlab) {
    "show ddos policer stats all"
    "show pfe statistics traffic"
    "show jnh host-path-stats"
    "show jnh 0 exceptions terse"
}

set vmx_re_cli(attga302me6@attlab) {
    "show system statistics"
}

set vmx_pfe_cli_hostpath(attga302me6@attlab) {
    "show ttp stats"
    "show packet statistics"
}

#project 2
set cmds2(attga302me6@attlab) [list            \
    vmx_re_cli                          \
    "start shell pfe network fpc0"      \
    vmx_pfe_cli                         \
    vmx_pfe_cli_hostpath                \
    exit                                \
]

3. introduction

An old topic

There were no good ways/tools to interact with a remote host/router from within a shell script…​the expect script (or other expect-like or expect-inspired tools) might be the best available tool that can help on this.

The goal

The goal of implementing this script, is to provide a generic method , to automate tasks that involves interactions with a remote host, being it a router, switch, firewall, a server, or even the local host itself.

The idea

The idea is to ``disassamble'' the interactive tasks into different stages or components, then implement those most common components in the form of some generic modules. once this is done, then most individual tasks can be composed by just reassambling one or more of these common modules, which optionally can be further fine tuned by a bunch of "config options" (or "knobs"). This can be implemented as parameterized command line interface (CLI), or as configuration command in a configuration file. the benefit is obvious - the script can be used either as an individual terminal based tool, or being called in a 3rd party script with different parameters as needed.

modules

Here are the most common modules in most of the interactive tasks:

The Modules	Functions
loggin process	login to the router with defined steps
sending commands(optional)	send a serial of commands and receive the outputs
(optional) monitoring outputs	examine the output of some commands for specific keyword(s)/value(s), maybe periodically
(optional) comparison	see if the monitored value(s) meet some pre-defined criteria
(optional) data collection	if yes, it is assumed the issue is seen ("reproduced"), then more information after the issue is reproduced
(optional) more test	if no, repeat some test over and over

options

options are all user-changeable attributes of a session, for example:

• anti_idle_timeout: for how long you would like to send a character to the session , just to prevent a telnet sesssion from being timed out?

• anti_idle_string: what string(s) you prefer to use for that purpose?

• max_round: how many iterations you would like to send those commands?

• interval_cmds: what’s the wait interval between each iteration of a batch of commands execution?

• interval_cmd: what’s the wait interval between each individual command?

• …​

3.1. some design considerations

problem in most small script

one problem in most small, "one-time use" script is that:it always assume a lot of things:

• presuming telnet is used as login prototocol,

• one level direct login ,

• prompt should be just ">",

• etc.

if the presumption changes next time the script won’t work - you need to understand the code and be able to change these attributes in order to make it work in other scenarios.

This script was designed with at least 2 things in mind:

configurable

• what kind of device you want to login ?(junos, junos-e, cisco, linux server, or even local machine…​)

• how do you want to login to a device? (telnet, ssh, ftp, rlogin, …​)

• how many level it takes to get to the target machine? (via 2 springboard, an a equipment list/menu and another login)

• user name, password, yes/no answers, menu selections

• what command you want to send right after the login process?

• do you even want to automate the command sending after login? or just stay interactive mode and type command by yourself?

One benefit is that all config options can be centralized in one seperate configuration file, so the user of the script can just change the script behavior and don’t need to read and understand the code at all.

re-use local existing resource

Use whatever tools avaiable in the local machine (ssh, telnet,ftp, rcp, sftp, your another scripts, etc). There is no presumption/limitation about what tools you want to use, Telnet/SSH clients are the most common ones though.

fully parameterized

all configurable options can be either specified in a centralized config file, or directly from the command line.

this makes the tool "script-friendly" and can be re-used conveniently in other scripts. This will very much leverage the power of UNIX CLI and make the further extension more efficient.

Some useful/interesting features will be highlighted below.

3.2. feature highlights

this small script currently supports most of the features I could expect from other commercial tools dealing with interactive sessions (e.g. secureCRT). ^[7] I may extend it whenever I feel other useful features/options can be easily implemented.

auto-login

you "program" the login steps in a config file, the script will follow the steps to login to remote device.

hide login details

this is to suppress the (maybe sensitive) login steps that normaly would be printed to the terminal (not in secureCRT?).

anti-idle

by default a session will be terminated if nothing has been typed in a certain period of time. this feature make it possible to retain the session by sending "keepalives" - usually a configurable ("control") character (anti_idle_string) after every certain period of time (anti_idle_timeout).

persistent session (or, "reconnect on disconnect")

as soon as the session got disconnected, the script will detect this event, reconnect, and continue either from where it left off, or start executing the commands from all over again (not in secureCRT?).

timestamping

a lot of devices don’t print a timestamp when a command was typed. the timestamp option make it possible to timestamp any typed command, or even all lines from command output, from any device it logged in. This might be useful when when testing time-sensitive issues [not in secureCRT?].

sending commands after login

to send some predefined commands, either right after a successful login, or anytime in the session.

"quickmode"

login, send commands, and then instead of staying interactive (the default) but just exiting. this is useful when used in a shell script which, needs to carry on after a command output is grabbed.

issue definition and replication

this maybe an interesting feature. Currently there are two options that one allows the user to provide a regex to catch some specific pieces of information from the command output, and the other one allows the user to provide an "logical expression" to define what exactly should be treated as an "issue". Depending on the evaluation of the "issue", the script will either optionally examine pre-defined actions and execute them if the issue is "seen", or execute some configured "test" before continuing to the next iteration. This can be used in problem replications.

email notification

sending an email notification is easy and cheap way to report issues to the user from a script. The receipee can be the current user or whoever configured (in the emailto option), when any of the defined criterias is met [not in secureCRT].

"autopaging"

with this knob turned on, the command "show config | no-more" for example can be simplied as just "show config". the script will monitor the command output and it will, once detected some pending content, keep pulling the rest of the output (by keep "pressing" spaces or other configurable keys for you) until hitting the end of the output [not in secureCRT].

Overall, this small script implemented:

• something like "the secureCRT", but …​

• in command line mode (so server/script-friendly) and …​

• was tailored with just those most interested and useful part of the features. (cutting off all GUI related features, e.g, the "font", "color" ,etc — who cares about the font in a terminal? ).

• plus something misc small features (email sending, timestamping, hide login, issue monitoring, etc).

so it is much smaller and more programmable, and can be running from a terminal of any unix/linux/mac machine equiped with a standard expect tool, which is the only software requirement.

Tip

The well known benefit of a CLI script implementation (vs. a GUI) is that, it is much easier to be integrated into other tools, being it a perl/shell/python/tcl script, or even another instance of the same script. All standard unix pipe/redirection/job management mechanism applies and all traditional terminal tools can be used.

4. how to run the script

As mentioned, the crtc tool runs on any *nix or *nix-like machine equiped with basic expect software. it does not require any additonal libraries/3rd party modules/etc. and as a small script basically there is nothing to "install" but simply run it from my folder.

All examples in this doc work in linux, freebsd, Mac, cygwin.

Note	the same file crtc includes all codes, documents [8], options default values (can be overided by options in config files or options in command line) , etc: tcl/expect codes (the script) default value of all config options usage, "online" document (this doc)

To run the script, follow one of these steps:

• just execute it directly from my folder:

  ~pings/bin/crtc -c "show chassis alarms" -n 3 -i 30 alecto@jtac

  Important

  this method does not work for "recursive" crtc execution - a "recursive" execution happens when one crtc script is calling another crtc script instance in the configured login or command data, which will be explained further later.

• add my script location to your system PATH variable, and execute crtc directly without referring where it is.

  pings@svl-jtac-tool02:~$ export PATH=$PATH:~pings/bin/
  pings@svl-jtac-tool02:~$ crtc REMOTEDEVICE

  Tip	this is recommended, since no need to copy and you are always use the updated version of the script.

• "install" crtc under your own folder

  if you perfer to run it under your home folders or wherever, just copy the 2 files over and chmod the script will be it.

  Tip	Optionally you can also change your PATH environment variable, to make the system to be able to locate this script from your folder. or put the PATH in your .bashrc to make it persistent.

  svl-jtac-tool02#cd YOUR_FOLDER
  (YOUR_FOLDER)#cp /homes/pings/bin/crtc ./
  (YOUR_FOLDER)#cp /homes/pings/bin/crtc.conf ~/
  (YOUR_FOLDER)#chmod 755 crtc
  (YOUR_FOLDER)#export PATH=$PATH:YOUR_FOLDER

  then run the script without the need to refer my folder:

  #crtc myserver

4.1. the script: just one file

here is the file in jtac server (e.g svl-jtac-tool02):

[pings@svl-jtac-tool02 ~]$
[pings@svl-jtac-tool02 ~]$ cd bin
[pings@svl-jtac-tool02 ~/bin]$ ls -l | grep crtc
-rwxr-xr-x   1 pings  others    5802 Nov 21 10:48 crtc      #<-the script
-rw-r--r--   1 pings  others   82313 Feb 16 20:20 crtc.conf #<-config file

as can be seen the script itself is just one file crtc , it will read a config file crtc.conf before start running.

4.2. config file

a config file is a place where you can define or fine-tune all of the data, options , or the default values of options, that will influence the behavior of the script.

The crtc script will try to locate a config file in below sequence:

• from what is specified in -C CONFIG_FILE_FULLNAME option

• from the same folder where crtc script is located

• from home folder

accordingly, it can be generated in one of the following ways:

1. create a config file manually

   • create a config file named crtc.conf and place it under your home folder.

   • if you would like to use a config file with other names or under other location, specify a -C CONFIG_FILE_FULLNAME command option when running crtc.

2. copy an existing template config file into your home folder

   cp ~pings/bin/crtc.conf ~/

3. generate a config template from crtc directly with -g option when executed:

   crtc -g

   this will generate a config file template crtc.conf.template, which can be renamed/modified as wished.

4.3. config options

the value of an option (another name - "config knob") can be changed at different time of a session, and at least in 5 different places (levels):

1. initial default value defined in crtc script when it got initiated

2. config file

3. environment variable

4. command line options

5. "inline" in the session (keystroke commands)

Tip	the keystroke commands is some "keystrokes" you can type in after the session started.

basically this is how the script runs and the sequence that a config option get its value:

1. start with a default "hard-coded" value in the crtc script itself

2. check and see if a config file exists (~/crtc.conf), or specified from command line (-C FILENAME), read the options setting from it once found

3. check if there is any environment variable configured for crtc (name started lik CRTC_NNN)

4. accept and reset options from command line

5. start to automate the login to the remote device with configued steps and send provided commands

6. once login succeed, return control to the user

7. from now on what the user can do:

   • type commands interactively to the device,

   • type some special "keystroke commands" to change the options

Taking an example of the timestamp option:

• the default, initial value is 0

• set timestamp 1 in config file will overide it to 1

• export CRTC_timestamp 0 from shell (before running crtc) will change it to 0

• now running crtc with option -t 0, will set the timestamp back to 1

• after crtc finish the login automation and return the control to you , press a "keystroke command" !t (a ! and a t , literally) will toggle the timestamp value to 0, type !t will toggle it again back to 1.

The same applies to other options.

5. crtc options

5.1. options list

To provide the most control to the login process, crtc current supports options and "inline" commands that can be used before, or/and after the login process begins. To get a list, run crtc with -h option:

ping@ubuntu47-3:~$ crtc -h

or, in a session type !h command, both will print a list with a short explanation of currently supported options and inline commands, :

ping@ubuntu47-3:~$ crtc -h
Usage:/home/ping/bin/crtc/crtc [OPTIONS]
  OPTIONS:
     -G                 :generate a config file template
     -h/H               :this usage/detail tutorial
     -K                 :kibitz(,not in use,TODO)
     -e                 :edit config file
     -l                 :list configured hosts info
     -v                 :print version/license info
Usage:/home/ping/bin/crtc/crtc [OPTIONS] session
  OPTIONS:
     -A                 :set/unset auto-paging (no-more)
     -a                 :set arbitrary attributes(options)
     -b/B "show cli"  :commands before/after loop
     -c "show version | no-more" :commands (in a loop)
     -C <config file|NONE> :read config file
     -d                 :set/unset debug info
     -D                 :max_hits: max times issue will be detected
     -e ">" -s "show version | no-more" :expect and send
     -E "$" -S "telnet alecto.." :same, but for login
     -f router1.log     :router log file name
     -F log_dir         :folder name of log file
     -g                 :gap between each cmd
     -h <host1> <host2> .. :login to multiple hosts
     -i <SECONDS>       :interval between each iteration of all cmds
     -j <NUMBER>        :project
     -J                 :event script
     -k                 :exit session also/not exit script
     -l "pings@juniper.net" :email address to send log attachment
     -L                 :email subject
     -m                 :monitor mode(keep sending cmds)
     -n <count>         :send command <count> time(s)
     -o                 :set/unset "login only" (ignore all cmds)
     -O                 :emailcont: email contant
     -p                 :set/unset "persist mode"
     -P                 :when used with -h,run commands in parallel
     -q                 :set/unset "quick mode" (quit after done)
     -Q                 :disable all "features"
     -r <SECONDS>       :reconnect interval
     -R "1@packets@s+(d+)s+(d+) pps@packets@pps" :regex and vars string
     -I "1@pps == pps_prev"  :issue definition
     -t                 :set/unset timestamping all commands
     -T                 :timeout all output lines
     -u                 :continue_on_reconnect
     -U                 :print "login_succeed_signature" when login succeeded
     -H                 :set/unset "hidden mode" (hide login step details)
     -w <SECONDS>       :waittime_login
     -W                 :use crtc in shell: set in_shell
     -V                 :print value matched by the regex from -R
     -x                 :all_med - define issue as all cmds met -I criterias
     -X                 :lock_session, set key_interact to an impossible value
     -y                 :use barewords instead of $var for varibles in -I expression
     -Y                 :commands when match found(Yes) in -R
     -N                 :commands when match not(No) found in -R
     -z                 :compress/don't compress log before attach to email
     -Z                 :no_anti_idle: disable anti_idle_timeout
to read the full manual:
     /home/ping/bin/crtc/crtc -H
inline commands:
  !a  toggle auto_paging option
  !b/B  not in use
  !c  specify and repeat a command
  !C  editing current config file
  !d  toggle debug option
  !D  dump (debugging) input/output characters
  !e!  start expect-command pair group automation
  !e?  same, but asking for array's name
  !E  attach log in email and send to user
  !f/F  not in use
  !g/G  not in use
  !H  toggle hideinfo option
  !I  enter interpreter mode
  !j/J  not in use
  !k  toggle exit_sync option
  !K  not in use
  !l log file operations
    !le: email current log file
    !ln: start a new log file
    !ls: stop/suspend logging to current log file
    !lS: stop/suspend all logging
    !lr: resume logging to current log file
    !lv: view the log file
    !ll: list the log file(name)
  !v(=!lv):view the log file
  !m  print host list
  !M  not in use
  !n  not in use
  !N  toggle no_feature
  !o  toggle login_only option
  !O  reload the config
  !p  toggle persistent option
  !P  toggle parallel option
  !q  toggle nointeract (quick mode) option
  !Q  exit the script
  !r  repeat the previous cmds execution
  !R  resume the previous cmds from where left over
  !s  stop the remaining of previous automations
  !S  not in use
  !t  toggle timestamp option
  !T  toggle timestamp_output_line option
  !u/U  not in use
  !v  print version/license info
  !V  not in use
  !w/W  not in use
  !x/X  not in use
  !y/Y  not in use
  !z  not in use
  !Z no_anti_idle
session management
  i
  N  N refers a session number
helps
  !?  list all keystoke commands
  !h  print usage
  !i  print detail tutor docs
other commands:
  Ctrl-g suspend the script
  Ctrl-(SIGQUIT) to stop the cmds automations
  automation control:
      q   quit the automation
      <SPACE> continue(escape sleeping)
      <ENTER> same as <SPACE>
      Q   exit the script
  SPECIAL CMDS:
     GRES               :Junos GRES request
     SLEEP <SECONDS>    :sleep some seconds before sending next command
     REPEAT M N         :repeat the last M commands N times
     UPGRADE /var/tmp/..:upgrade junos release

Note	there is one rule here when running crtc with options: the session name must be the last parameter in the command line.

5.2. option maps

TODO

CLI	inline cmd	options	options_cli index (internal use)	data structure(internal use)
-A		auto_paging	auto_paging
-B			post_commands	post_cmds_cli
-b			pre_commands	pre_cmds_cli
-c	!c		commands	cmds1
-C		config_file	config_file
-d		debug	debug
-D		max_hits	max_hits
-e			expect	cmds_cli
-E			EXPECT	login_info_cli
-F		log_dirname	log_dirname
-f		log_filename	log_filename
-g		interval_cmd	interval_cmd
-H		hideinfo	hideinfo
-h			hosts	hostlist
-i		interval_cmds	interval_cmds
-I			issue	issue_info
-k		exit_sync	exit_sync
-K		kibitz	kibitz
-L		emailsub	emailsub
-l		emailto	emailto
-m		max_rounds 1000000000	max_rounds 1000000000
-n		max_rounds	max_rounds
-N			reproduced_no	test_cli
-O		emailcont	emailcont
-o		login_only	login_only
-P		parallel	parallel
-p		persistent	persistent
-q		nointeract	nointeract
-Q	!N	feature	feature
-r		reconnect_interval	reconnect_interval
		reconnect_on_event	reconnect_on_event
-R			regex_vars	regex_info
-s
-S
-t		timestamp	timestamp
-T	!T	timestamp_output_line	timestamp_output_line
-u		continue_on_reconnect	continue_on_reconnect
-U			CONTINUE_ON_RECONNECT
-V		print_matched_value	print_matched_value
-v		version	version
-w		waittime_login	waittime_login
-x		all_met	all_met
-X		lock_session	lock_interact
-y		easyvar	easyvar
-Y			reproduced_yes	collect_cli
-z		compress_log	compress_log
-Z	!Z		no_anti_idle

5.3. internal data structure (arrays)

5.3.1. login_info

set login_info(myrouter)       [list               \
    "$"        "telnet alecto.$domain_suffix"    \
    "login: "      "lab"                        \
    "Password:"    "lab123"                   \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"          \
]

this is used to store all information needed to login to a remote device.

Note

due to the tcl method of handling regex, if "$" doesn’t work, use "\\\$". see appendix for more details on this. $ MAY or may NOT work: set login_info(myrouter) [list \ "$" "ssh myjumpstation1" \ "password" "mypass" \ "$" "ssh my_target_machine" \ ">" "set cli timestamp" \ ] \\\$ will work: set login_info(myrouter) [list \ "$" "ssh myjumpstation1" \ "password" "mypass" \ "\\\$" "ssh my_target_machine" \ ">" "set cli timestamp" \ ]

5.3.2. cmdsN

the N in cmdsN can be between 1 to 10, this is used to store all commands to be sent to the remote device AFTER login was successful. all commands listed in this array will be executed, unless -o command line option or login_only config option is set, in that case crtc will just login and data in cmdsN array won’t be executed. the cmdsN array can be executed multiple time, if the max_rounds were configured to a number bigger than one.

5.3.3. pre_cmdsN

same as cmdsN, but will be executed just 1 time BEFORE the cmdsN array.

5.3.4. post_cmdsN

same as pre_cmdsN, but will be executed just 1 time AFTER the cmdsN loop.

5.3.5. regex_info

5.3.6. issue_info

5.3.7. collect

contains a list of commands that will be sent if the expected issue appears

5.3.8. testN

contains a list of commands that will be sent if the expected issue didn’t appear.

Note

you can also put all commands to be sent in the same login_info array. set login_info(myrouter) [list \ "$" "telnet alecto.$domain_suffix" \ "login: " "lab" \ "Password:" "lab123" \ ">" "set cli screen-width 300" \ ">" "set cli timestamp" \ ">" "configure" \ "#" "set interfaces xe-3/1/0 disable" \ "#" "show | compare" \ "#" "commit" \ ] but put in a seperate array `cmds1` is better. the `login_info` was used mainly for login process, attach all commands in same array will make the `login_only` (just login and ignore all later commands) option useless. another way to send commands after successful login is through command line parameters (or options), which will be introduced later in this doc.

5.3.9. an example

Here is an example illustrating the usage of these internal arrays.

for example, right after logged in to the router, you want to:

• flap a port 3 times

• with 10s interval between each iteration

• in each iteration:

  • disable the port

  • commit the config change

  • wait 3s before bringing it back up

  • rollback config to bring the port back

  • commit the config change

these actions can be configured in a cmdsN array, e.g cmds1:

set cmds1(myrouter) {"configure" "set interfaces xe-3/1/0 disable" commit "SLEEP 3" rollback 1 "show | compare" commit exit}

you may not like to enter configure (to enter "privilidge mode") and exit (to exit the "privilidge mode") commands over and over again, and it maybe better to enter "privilidge mode" once, from there to repeat those port flap actions and then exit once the test is done.

To achieve that:

• put configure in pre_cmds1

• put exit in post_cmds1

so the config looks:

set pre_cmds1(myrouter) configure
set cmds1(myrouter) {"set interfaces xe-3/1/0 disable" commit "SLEEP 3" rollback 1 "show | compare" commit}
set post_cmds1(myrouter) exit

The 2nd command list looks too long, you can use a continuation sign \ to cut it shorter and make it one command per line.

set pre_cmds1(myrouter) configure
set cmds1(myrouter)         {
    "set interfaces xe-3/1/0 disable"
    "commit"
    "SLEEP 3"
    "rollback 1"
    "show | compare"
    "commit"
}
set post_cmds1(myrouter) exit

if you would like to do the same test often, but may need to change some parameters to different values sometime, you can use "variable substitutions" in your commands. This can be done via the list command, with which you can includes variables in commands, the format is:

[list "command1" "commands2 with $param", "command3" ...]

now the commands list can be written to this:

set sleeptime 3
set port "xe-3/1/0"

set pre_cmds1(myrouter) configure

set cmds1(myrouter)         [list   \
    "set interfaces $port disable"   \
    "commit"   \
    "SLEEP $sleeptime"   \
    "rollback 1"   \
    "show | compare"   \
    "commit"   \
]

set post_cmds1(myrouter) exit

now run the crtc script and the above commands will be executed.

~pings/bin/crtc myrouter

Just changing the varible "sleeptime" and "port" if different values are needed for a new test.

the other internal arrays (regex_info, issue_info, collect and testN) will be illustrated later.

5.4. user_patterns

TODO:

user_patterns array can be used to configure the script behavior when the user defined messages appear. it can be used either under automation/batch mode when crtc is still iterating the command sending, or under interact mode when user get the control.

some considerations about the data structure designed:

• user_patterns(router) introduce complexity - every device need to re-define the same

• use one list, but use 2nd element as an action, making it looks like pattern action pair, so existing code may be used

the value could be:

• list of 1 pattern:

  • a single-instance pattern: [list "ould not resolve"]

  • a multi-instance pattern: [list "ould not resolve|bla bla"]

• list of 1 pattern + 1 action, where action can be:

  • a real command or string to send: [list {\(no\)} yes]

  • a special command:

    • RETRY: [list "error: Backup RE not running" RETRY]

    • RECONNECT: [list "Connection refused" RECONNECT]

• list of 1 pattern + 1 actions + optional attributes

  • set user_patterns(nsr_not_active) \

    [list "warning: GRES not configured" cmds3 interact]
          ------------------------------ ----- --------

    user defined pattern(s)              this applies to interact
                                         also

    command
    command group or
    special command:
      RETRY RECONNECT EXIT ...

TODO: there might be "overlaps" between tests, e.g. below pattern was configured for GRES, but it may appear in both GRES and ISSU:

#set user_patterns(gres_success) [list "routing engine becomes the master"]

current workaround is to simply comment it out when doing ISSU test.

Warning

this will have "performance impact" as it slow down the whole command output process - this will be an issue only when there is VERY long output, e.g.: "show config | no-more" in that case set attribute: -a "set enable_user_patterns 0"

6. more usage examples

6.1. monitoring: keep sending cmd(s) in a loop

supposing you need to monitor some datapoints from a remote device, not one or two shot, but periodically , or even continuously. keep repeating the command manually is boring. the -n option specifies how many iterations you want to repeat the same cmd(s), and -i specified the interval between iterations. just like the way the windows ping works with its -n and -i options.

in a shell script, you might want to get the output of some command from the remote device in realtime, and then contiue the shell script with the collected data. unless you are scripting directly within the router (.e.g. Junos unix shell mode), this involves at least the following actions:

• login to the router (telnet/ssh , username, password, sprintboard, etc)

• send one or more commands (-c, or -e -s)

• repeat it 3 times, with 5s interval between each iteration (-n -i)

• collect the output for your analysis

• trim other garbage texts (e.g username/pass/warning/annotation/etc) (-H)

• disconnect once above task done and run next shell command (-q)

here is how things can be done with this tool, in the form of a "one-liner":

Example 15. use -n N -i INTERVAL to monitor alarms

ping@ubuntu1404:~/bin$ crtc -c "show system alarm" -n 3 -i 5 -q -H alecto
current log file ~/logs/alecto.log
set cli timestamp

Dec 01 13:30:32
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0> iteration:1
show system alarm
Dec 01 13:30:32s
2 alarms currently active
Alarm time               Class  Description
2014-11-24 08:58:27 EST  Minor  PEM 1 Absent
2014-11-24 08:58:27 EST  Minor  PEM 0 Absent

{master}
lab@alecto-re0> iteration:2
show system alarm
Dec 01 13:30:37s
2 alarms currently active
Alarm time               Class  Description
2014-11-24 08:58:27 EST  Minor  PEM 1 Absent
2014-11-24 08:58:27 EST  Minor  PEM 0 Absent

{master}
lab@alecto-re0> iteration:3
show system alarm
Dec 01 13:30:42s
2 alarms currently active
Alarm time               Class  Description
2014-11-24 08:58:27 EST  Minor  PEM 1 Absent
2014-11-24 08:58:27 EST  Minor  PEM 0 Absent

{master}
lab@alecto-re0> bye!:)
ping@ubuntu1404:~/bin$

Example 16. to monitor interface counter and ospf neighborship

without -q, the session won’t exit after the script complete all iterations. you can continue typing commands manually.

ping@ubuntu1404:~/bin$ crtc -c "show interfaces xe-3/1/0 | match \"input packets\"" -c "show ospf neighbor" -H -n 3 -i 5 alecto@jtac
current log file ~/logs/alecto.log
iteration:1
set cli timestamp

Dec 01 14:06:29
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0> show interfaces xe-3/1/0 | match "input packets"
Dec 01 14:06:29
    Input packets : 13790950

{master}
lab@alecto-re0> iteration:2
show ospf neighbor
Dec 01 14:06:29
Address          Interface              State     ID               Pri  Dead
10.192.0.41      xe-3/1/0.0             Full      192.168.0.6      128    37
10.192.0.45      xe-4/1/0.0             Full      192.168.0.7      128    38

{master}
lab@alecto-re0> show interfaces xe-3/1/0 | match "input packets"
Dec 01 14:06:34
    Input packets : 13790960

{master}
lab@alecto-re0> iteration:3
show ospf neighbor
Dec 01 14:06:34
Address          Interface              State     ID               Pri  Dead
10.192.0.41      xe-3/1/0.0             Full      192.168.0.6      128    32
10.192.0.45      xe-4/1/0.0             Full      192.168.0.7      128    33

{master}
lab@alecto-re0> show interfaces xe-3/1/0 | match "input packets"
Dec 01 14:06:40
    Input packets : 13790964

{master}
lab@alecto-re0> it's all yours now :)

-m

this can be shortened by -m option, the cmds will be sent endless time, with whatever interval value configured in the config file. if the interval was not configured then the default value 0 will be used, meaning to send as fast as the router can response.

set interval 0

login:

same as above, but just execute the cmd(s) endlessly.

6.2. group multiple options

crtc support compact option list, so the following 3 commands are the same:

crtc -c "show system uptime" -q -H -n 3 -i 5 alecto@jtac
crtc -c "show system uptime" -qHn 3 -i 5 alecto@jtac
crtc -c "show system uptime" -qHn3 -i5 alecto@jtac

this makes the command options shorter to type.

I tried to make the options compatible as unix conventions (see some interesting references:), but I don’t find a good "existing parser" that I can just borrow, the "cmdline" package doen’t look good, plus I don’t want to introduce any external dependencies - I have to implement a cmdline parser myself.

6.3. some "special" commands

6.3.1. GRES command (Junos platform specific)

when "sending" a "GRES" command, what crtc does under the scene is:

• to interact with Junos router to perform a real JUNOS GRES operation.

• reconnect to the router whenever connection got bounced.

This will be useful for GRES related test.

Tip	The command "GRES" itself will never get sent literally to the router at all.

to request GRES to a dual-RE Junos device, simply use the "fake" GRES command:

crtc -c GRES alecto

this will invoke Junos request chassis routing-engine master switch command and "press" 'yes' to proceed the RE switchover procedure. the session will be disconnected as expected , and the script will detect this and also exit .

Example 17. "GRES" command will trigger GRES operation and make the crtc script exit

ping@ubuntu1404:~/bin$ crtc -c GRES alecto
current log file ~/logs/alecto.log
telnet alecto.jtac-east.jnpr.net
ping@ubuntu1404:~/bin$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re0> iteration:1
set cli timestamp

Dec 01 19:02:31
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0>
Dec 01 19:02:31

{master}
lab@alecto-re0> request chassis routing-engine master switch
Dec 01 19:02:31
Toggle mastership between routing engines ? [yes,no] (no) yes
Dec 01 19:02:32

Resolving mastership...
Complete. The other routing engine becomes the master.

{backup}
lab@alecto-re0> it's all yours now!:)
exit
Dec 01 19:02:32

Connection closed by foreign host.

6.3.2. "GRES" with "persistent" session

if this is all what you wanted, that’s fine. but being disconnected means you have to restart the session again, and if the router is not ready yet, you will have to wait for a while and retry - all these should/can be handled by the script:

• detect if the remote device is reachable and

• if yes, re-login automatically.

• if not, wait for while and keep detecting

  Feb 27 15:17:04
  Command aborted. Not ready for mastership switch, try after 229 secs.

  labroot@alecto-re0> [crtc:ops! sorry...will redo  230s later then...]
  [crtc:will count 230 seconds and retry...]

  <<<<count {200}s before proceeding...
  <<<<type anything to skip...

  <<<<count {30}s before proceeding...
  <<<<type anything to skip...

  labroot@alecto-re0> request chassis routing-engine master switch
  Feb 27 15:20:54
  warning: Traffic will be interrupted while the PFE is re-initialized
  Toggle mastership between routing engines ? [yes,no] (no) yes

  Feb 27 15:20:54
  Resolving mastership...
  Complete. The other routing engine becomes the master.
  will reconnect in 10s

Example 18. "GRES" command with -p option will trigger GRES and then relogin the router automatically

ping@ubuntu1404:~/bin$ crtc -pc GRES alecto
current log file ~/logs/alecto.log
telnet alecto.jtac-east.jnpr.net
ping@ubuntu1404:~/bin$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re0> set cli timestamp

Dec 01 19:41:36
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0>
Dec 01 19:41:36

{master}
lab@alecto-re0> request chassis routing-engine master switch
Dec 01 19:41:37
Toggle mastership between routing engines ? [yes,no] (no) yes
Dec 01 19:41:38

Command aborted. Not ready for mastership switch, try after 11 secs.

{master}
lab@alecto-re0>
Dec 01 19:41:50

{master}
lab@alecto-re0> request chassis routing-engine master switch
Dec 01 19:41:50
Toggle mastership between routing engines ? [yes,no] (no) yes
Dec 01 19:41:51

Resolving mastership...
Complete. The other routing engine becomes the master.

{backup}
lab@alecto-re0> it's all yours now!:)
exit
Dec 01 19:41:51

Connection closed by foreign host.

will reconnect in 30s
ping@ubuntu1404:~/bin$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re1 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re1> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re1> set cli timestamp

Dec 01 19:42:24
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re1>

there is a tricky part in Junos GRES - if you do it too frequent, say, do it again in less than 4m after the previous execution, the router will reject the request with a message. This tool can handle this situation well:

Example 19. hold and retry when GRES request was rejected

ping@ubuntu1404:~$ crtc -c GRES alecto
current log file ~/att-lab-logs/alecto.log
telnet alecto.jtac-east.jnpr.net
ping@ubuntu1404:~$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re0> set cli timestamp

Nov 23 01:36:24
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0>
Nov 23 01:36:24

{master}
lab@alecto-re0> request chassis routing-engine master switch
Nov 23 01:36:25
Toggle mastership between routing engines ? [yes,no] (no) yes
Nov 23 01:36:26

Command aborted. Not ready for mastership switch, try after 221 secs.                                       #<------

{master}
lab@alecto-re0> [Sun Nov 23 01:35:52 EST 2014::..[script:ops! sorry...will redo 221s later then...]..]      #<------
Nov 23 01:42:50

{master}
lab@alecto-re0> request chassis routing-engine master switch                                                #<------
Nov 23 01:42:50
Toggle mastership between routing engines ? [yes,no] (no) yes
Nov 23 01:42:51

Resolving mastership...
Complete. The other routing engine becomes the master.

{backup}
lab@alecto-re0> [Sun Nov 23 01:42:18 EST 2014::..[great! will exit current session...]..]
exit
Nov 23 01:42:51
Connection closed by foreign host.
[Sun Nov 23 01:42:18 EST 2014::..will reconnect in 30s..]
ping@ubuntu1404:~$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re1 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re1> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re1> set cli timestamp

Nov 23 01:43:24
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re1> [Sun Nov 23 01:42:50 EST 2014::..it's all yours now!:)..]

Nov 23 01:43:24

{master}
lab@alecto-re1>

Note	if you are trying to switchover too frequently (less than 4min?) , by default Juniper router will reject the GRES request and pop out a complaining message. the "GRES" procedure in this script currently can parse the message and find out the right seconds to hold before next attemp.

6.3.3. repeat the "GRES" operation multiple iterations

the following command will perform GRES 10 times, with 300s intervals and reconnect after 10s after each switchover.

crtc -c GRES -pn3 -i 10 -r 10 alecto@jtac

this might be useful in the GRES related test/bug replications.

6.3.4. SLEEP command (platform-independent)

config

NONE

Example 20. SLEEP command: set intervals between each command

sending a SLEEP 10 "pseudo" command will actually send nothing to the remote device, but just to slow down the next command with the specified number of second. So this is useful when different intervals need to be set between each command.

ping@ubuntu1404:~/bin$ crtc -c "show system alarm" -c "show system uptime" -c "SLEEP 10" -c "show system alarm" alecto

current log file ~/logs/alecto.log
telnet alecto.jtac-east.jnpr.net
ping@ubuntu1404:~/bin$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re1 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re1> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re1> set cli timestamp

Dec 01 22:15:20
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re1> show system alarm
Dec 01 22:15:21s
3 alarms currently active
Alarm time               Class  Description
2014-12-01 19:42:06 EST  Minor  PEM 1 Absent
2014-12-01 19:42:06 EST  Minor  PEM 0 Absent
2014-12-01 19:41:55 EST  Minor  Backup RE Active

{master}
lab@alecto-re1> show system uptime
Dec 01 22:15:21
Current time: 2014-12-01 22:15:21 EST
System booted: 2014-08-24 17:55:41 EDT (14w1d 05:19 ago)
Protocols started: 2014-12-01 19:41:51 EST (02:33:30 ago)
Last configured: 2014-12-01 19:37:52 EST (02:37:29 ago) by root
10:15PM  up 99 days,  5:20, 1 user, load averages: 0.32, 0.22, 0.09

{master}
lab@alecto-re1> it's all yours now!:)
show system alarm
Dec 01 22:15:31s
3 alarms currently active
Alarm time               Class  Description
2014-12-01 19:42:06 EST  Minor  PEM 1 Absent
2014-12-01 19:42:06 EST  Minor  PEM 0 Absent
2014-12-01 19:41:55 EST  Minor  Backup RE Active

{master}
lab@alecto-re1>
Dec 01 22:15:31

{master}
lab@alecto-re1>

6.4. integration with other tools in shell

6.4.1. work with shell script

here is an example of using crtc in shell script:

Example 21. interact with remote device within shell

file: printver.sh

#!/bin/bash
ver=`crtc -Hqc "show version | no-more" $1 | grep -i "base os boot" | awk '{print \$5}'`
echo "router $1 is running software versoin: $ver"

run the shell script:

ping@ubuntu1404:~/bin$ printver.sh alecto@jtac
router alecto@jtac is running software versoin: [12.3-20140210_dev_x_123_att.0]

ping@ubuntu1404:~/bin$ printver.sh tintin@jtac
router alecto@jtac is running software versoin:  [12.3R3-S4.7]

another good example is to "sweep" or "scan" a subnet and find out all login-able junos device, then report the requested data:

Example 22. scan all routers in a subnet

use -w to specify a relatively lower "patience" . This is the amount of time the script is willing to wait for a response from the remote device after the connnect attemp was started. with -w 5, if there is no response within 5s (say, a prompt asking for username), the script will stop and quit.

this can be used to quickly probe all IPs within a subnet.

for i in {1..254}
do
    echo "probing 172.19.161.$i ..."
    crtc -Hqac "show chassis hardware" -w 5 172.19.161.$i@jtac
done

or a oneliner for short:

for i in {1..254}; do crtc -Hqac "show chassis hardware" -w 5 172.19.161.$i@jtac; done;

6.4.2. work with GNU screen/tmux

if you are familiar with any of the terminal "multiplexer" kind of tools (GNU screen, tmux, byobu, etc), you can now run the crtc script within those tools.

with GNU screen/tmux , you can login to a server, run your program (crtc in this case), then shutdown your own PC and go home, while the session/test keep running in the server.

[pings@svl-jtac-tool02 ~]$ screen -fn -t alecto ~pings/bin/crtc myrouter

now you don’t need to remain the connection from your own PC (mostly windows) to your unix server where crtc is running.

ctrl-d to detech the screen session,

[pings@svl-jtac-tool02 ~]$ screen -ls
There is a screen on:
        11858.ttyp0.svl-jtac-tool02     (Detached)
1 Socket in /tmp/screens/S-pings.

You can leverate all power from GNU screen to manage the sessons.

6.5. send log as email attachment

TODO

6.6. other misc options

• -v: print version info

• -l: print all currently configured hosts

• -h: print some dummy help info

• -d: enable debug option

  in "debug mode" the script will throw out a lot more garbage info, and it’s mainly just for debugging/script developping purpose.

6.7. activate features on the fly

login and start command automations (as shown earlier)

pings@PINGS-X240:~$ crtc -c "show chassis alarms" -H -q -n 30 -i 10 alecto@jtac
no file /etc/resolv.conf exists!
log file: ~/logs/alecto@jtac.log
start to login alecto@jtac ...

crtc will start sending commands (-c) right after successful login:

execute cmds1 after login alecto@jtac!

      show chassis alarms
Mar 13 23:24:39
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> iteration:2=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 13 23:24:39
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

after each iteration, it will count 10 seconds before proceeding:

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...

timed out (10s) without user interuption...

   iteration:3=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 13 23:24:50
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...
 you hit something ,escape sleep (10s)...
iteration:4=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 13 23:24:52
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...

now, if you are monitoring the commands executions and wait to expedite it, just hit SPACE or ENTER to interupt the waiting period, crtc will escape the rest of the waiting seconds and execute the next iteration.

 you hit something ,escape sleep (10s)...
iteration:5=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 13 23:24:53
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...
you hit something ,escape sleep (10s)...
iteration:6=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 13 23:24:55
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...
you hit something ,escape sleep (10s)...
iteration:7=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 13 23:24:56
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

then, if for some reason you want to temperarily suspend the automation and type something else manually, just press ENTER 2 or 3 times quickly, the automation will be "suspended" and the control will be returned back to you now.

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...
you hit something ,escape sleep (10s)...
iteration:8=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 13 23:24:57
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

you want to type sth here? go ahead...      #<------you typed quickly at least 2 ENTER
it's all yours now!:)
 log file: ~/logs/alecto@jtac.log, ctrl-g to move to backgroud
 !t toggle local timestamp, !L peek log, !E to send log in email
 !h list configured hosts,  !? for more commands, !i for the full tutor doc

now the control is yours, you can type your other commands here:

lab@alecto-re1> show system uptime
you have unfinished automations (stack 1)!
 press !R to continue, !r to restart, ^\ or !s to stop!

Mar 13 23:58:35
Current time: 2015-03-13 23:58:35 EDT
System booted: 2014-08-24 17:55:24 EDT (28w5d 06:03 ago)
Protocols started: 2015-03-01 23:50:33 EST (1w4d 23:08 ago)
Last configured: 2015-03-08 22:54:00 EDT (5d 01:04 ago) by lab
11:58PM  up 201 days,  6:03, 2 users, load averages: 0.02, 0.04, 0.00

after every commands or ENTER you typed, crtc will prompt you that you still have some suspended automation task.

lab@alecto-re1>
you have unfinished automations (stack 1)!
 press !R to continue, !r to restart, ^\ or !s to stop!
Mar 13 23:58:38

lab@alecto-re1>

if you want to continue with whatever left off in the previous automation, press !R to resume it:

lab@alecto-re1> !R
  resuming previous automations ...
   show chassis alarms
Mar 14 00:00:53
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...
timed out (10s) without user interuption...
iteration:9=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 14 00:01:04
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

these operations can be repeated. For example, if you want to again suspend the automation, quickly type some ENTER will do it:

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...
you hit something ,escape sleep (10s)...
iteration:10=>alecto@jtac:
{show chassis alarms}
you want to type sth here? go ahead...
it's all yours now!:)
 log file: ~/logs/alecto@jtac.log, ctrl-g to move to backgroud
 !t toggle local timestamp, !L peek log, !E to send log in email
 !h list configured hosts,  !? for more commands, !i for the full tutor doc

you have unfinished automations (stack 1)!
 press !R to continue, !r to restart, ^\ or !s to stop! chassis alarms
Mar 14 00:01:06
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1>
you have unfinished automations (stack 1)!
 press !R to continue, !r to restart, ^\ or !s to stop!
Mar 14 00:05:17

if you decide to stop the prevoius automation, press !s or q:

lab@alecto-re1> !s - stop unfinished automations!

Mar 14 00:05:21

lab@alecto-re1>
Mar 14 00:05:22

lab@alecto-re1> !R
there is no automation to continue...

Mar 14 00:05:27

if you later want it back again, press !r to restart it:

lab@alecto-re1> !r - to repeat the previous cmds1 executions ,right((y)es/(q)uit?[y])
y
show chassis alarms
Mar 14 00:07:05
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> iteration:2=>alecto@jtac:
{show chassis alarms}
show chassis alarms
Mar 14 00:07:05
4 alarms currently active
Alarm time               Class  Description
2015-03-01 23:51:41 EST  Major  FPC 4 PIC 0 Failure
2015-03-01 23:50:34 EST  Minor  PEM 1 Absent
2015-03-01 23:50:34 EST  Minor  PEM 0 Absent
2015-03-01 23:50:33 EST  Minor  Backup RE Active

lab@alecto-re1> count {10}s before proceeding...
type anything to interupt...

or if you want to start another new automation (with a different command), press !n then answer the questions asked by crtc

lab@alecto-re1> !n
Enter the command array you configured:
    show system uptime
Enter how many iterations you want to run: [current 1]
    1000
Enter intervals between each iteration: [current 0]
    5
will iterate command [show system uptime] 1000 rounds with interval 5 between each iteration, (y)es/(n)o/(q)uit?
    y
show system uptime

new automation will start then:

Mar 15 00:52:24
Current time: 2015-03-15 00:52:24 EDT
System booted: 2014-08-24 17:55:24 EDT (28w6d 06:57 ago)
Protocols started: 2015-03-01 23:50:33 EST (1w6d 00:01 ago)
Last configured: 2015-03-08 22:54:00 EDT (6d 01:58 ago) by lab
12:52AM  up 202 days,  6:57, 2 users, load averages: 0.00, 0.11, 0.13

lab@alecto-re1> iteration:2=>alecto@jtac:
{show system uptime}
show system uptime
Mar 15 00:52:25
Current time: 2015-03-15 00:52:25 EDT
System booted: 2014-08-24 17:55:24 EDT (28w6d 06:57 ago)
Protocols started: 2015-03-01 23:50:33 EST (1w6d 00:01 ago)
Last configured: 2015-03-08 22:54:00 EDT (6d 01:58 ago) by lab
12:52AM  up 202 days,  6:57, 2 users, load averages: 0.00, 0.11, 0.13

lab@alecto-re1> count {5}s before proceeding...
type anything to interupt...
qyou stopped the automation!

Mar 15 00:52:30

6.8. running crtc without options/parameters

even if you don’t need to login to any remote device, just run crtc without any parameter will provide you some useful features. essentially what crtc does is to spawn a local bash according to your server’s default setting (whatever in environment variable $env(SHELL)). you can work in it as if nothing happened, plus you now get features like logging, command timestamping, listing configured hosts, etc. whenever needed, you can start another crtc instance to login to a remote device. the worst case - you won’t lose anything.

[pings@svl-jtac-tool02 ~]$ crtc
current log file ~/logs/LOCALHOST.log
it's all yours now!:)
date

pings@svl-jtac-tool02:~$ date
Wed Dec  3 05:56:46 PST 2014
pings@svl-jtac-tool02:~$            #<------you are now in a new shell

pings@svl-jtac-tool02:~$ !l         #<------list hosts configured in your config file
host list: qfx10@att LOCALHOST qfx11@att LOCALHOST@jtac vmx DT405JVPE alecto dt401-host vmx-vpfe qfx9@att LOCALHOST@att att-term-server2 e320-svl tintin myrouter

pings@svl-jtac-tool02:~$
[pings@svl-jtac-tool02 ~]$ pwd      #<------type your commands like normal
/homes/pings
pings@svl-jtac-tool02:~$ !t         #<------add timestamps
timestamp 1
pings@svl-jtac-tool02:~$ pwd
Dec 03 06:12:23 2014(local)
/homes/pings
pings@svl-jtac-tool02:~$ crtc alecto@jtac   #<------call another crtc to login to a router
Dec 03 06:00:51 2014(local)
current log file ~/logs/alecto.log
telnet -K  alecto.jtac-east.jnpr.net
pings@svl-jtac-tool02:~$ telnet -K  alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login: lab
Password:

--- JUNOS 12.3-20140210_dev_x_123_att.0 built 2014-02-10 20:40:15 UTC
{master}
lab@alecto-re0> set cli screen-width 300
Screen width set to 300

{master}
lab@alecto-re0> it's all yours now!:)
set cli timestamp

Dec 03 09:00:54
CLI timestamp set to: %b %d %T

{master}
lab@alecto-re0> exit                #<------exit the remote router, this will also exit the locally spawned new shell
Dec 03 09:14:26
Connection closed
                 [pings@svl-jtac-tool02 ~]$

the last exit command showed above will end up with two things happen: * exit the telnet session (initiated by the 2nd crtc instance) * exit the the first spawned shell (initiated by the 1st crtc instance)

this is because by default the option exit_sync was set. with this option the crtc script is able to detect the connection close event and exit local script. if this is not what you prefered, this can be changed at least with 3 ways:

• setting the config option exit_sync to 0

• use -K command option when running crtc

• hit !k key after run crtc

6.8.1. start new automations AFTER the initial automation finishs

one possible scenario that this might be useful is that, if you can’t automate the login process, but would like to automate the command sending after successful login. In a lot of (most) production networks a random token need to be typed in (possibly along with some "PIN" numbers) in the middle of the login steps. You can start crtc without options, login remote routers manually, then start the automation by typing one of the supported keystrokes (will add this later), which will trigger the new automation process.

t.b.c. examples:

• start crtc without options

  ~pings/bin/crtc

• login to the device manually (start ssh, input token, pin, password, etc)

  $ssh user@sprintboard
  please read your token and input number displayed on it: *******
  please input your personal PIN: ****
  ###############
  ###!welcome!###
  ###############
  secured-router >

• configure your new automations in the config file like this

  set cmds3(LOCALHOST)         [list      \
          ospf_commands                   \
          bgp_commands                    \
  ]

  set cmds3(LOCALHOST)         [list      \
          ospf_commands                   \
          system_commands                    \
  ]

  set ospf_commands(LOCALHOST) [list      \
      "show ospf neighbor"                \
      "show ospf interfaces"              \
  ]

  set system_commands(LOCALHOST) [list       \
      "show system uptime"                  \
      "show system alarms"             \
  ]

• now after you’ve worked on the session for a while and decide to start another automation to collect OSPF and BGP info from the router, just type !c!, literally . once crtc script see that, it will check cmds3 and start to execute all configured commands in it.

Note

in this example the script was started without a session/host name, so a "LOCALHOST" was used as the key (or "index") when composing the cmds3 array. for a session logged into a remote host, use the session name (always same as host name) to compose the array. command: ~pings/bin/crtc alecto@jtac configuration: set cmds3(alecto@jtac) [list \ ospf_commands \ system_commands \ ] set ospf_commands(alecto@jtac) [list \ "show ospf neighbor" \ "show ospf interfaces" \ ] set system_commands(alecto@jtac) [list \ "show system uptime" \ "show system alarms" \ ] to execute above configured commands, type: !c!

6.8.2. organize commands with nested command groups

as you maybe already noticed, the commands can be defined in a "nested" fasion, this make it easier to group your commands according to whatever criterias you prefered. You can organize your commands to different "levels" or "layers" like this:

set cmds3(yourdevice)   [list   \
        routing_protocols       \
        hardware_check          \
        snmp_check              \
        traffic_monitor         \
]

set routing_protocols   [list   \
        ospf_check              \
        bgp_check               \
        multicast_check         \
]

set hardware_check [list        \
    show chassis hardware       \
    show chassis alarms         \
]                               \

set snmp_check [list            \
    ......                      \
]                               \

set traffic_monitor [list       \
    ......                      \
]                               \

set ospf_check      [list       \
        show ospf neighbors     \
        show ospf interfaces    \
]                               \

set bgp_check   [list           \
        show bgp summary        \
        show bgp neighbors      \
]                               \

set multicast_check [list       \
        ...                     \
]                               \

......

6.9. commands "on the fly" (keymaps)

keystroke commands are the keys you typed in after the session control was returned back to you, after the login and commands automations. Not like the in GUI tools, where you can select the menus and buttons anytime to activate or start some specific features, in crtc everything you typed in was read and matched by the script itself, to compare with the pre-defined internal keybinds ,which was used as triggers to some features. ^[9]

here are some of the useful keybings.

6.9.1. !t !T timestamp every commands

when hitting !t, the timestamp feature can be toggled. when turned on, this feature will send a local timestamp on every carrige return, same as what Junos set cli timestamp knob does.

6.9.2. !c!, !e!

these keybinds can be used to start new automations.

6.9.3. other key commands (t.b.c)

• !a: toggle auto_paging option

• !e edit config file

• !l list currently configured hosts

• !k: toggle exit_sync option

• !o: toggle login_only option

• !p: toggle persistent option

• !q: toggle nointeract option (quick mode)

• !t: toggle timestamp option

• !u or !H toggle hideinfo option

• !n t.b.c

the full list of currently supported keybindings can be displayed by !?:

{master}
lab@alecto-re0>
Mar 13 23:24:57

{master}
lab@alecto-re0> !?key commands:
    !?  list all keystoke commands"
    !a  toggle auto_paging option"
    !c!  start commands group automation"
    !c?  same, but asking for array's name"
    !C  editing current config file"
    !d  toggle debug option"
    !e!  start expect-command pair group automation"
    !e?  same, but asking for array's name"
    !E  attach log in email and send to user"
    !h  print usage"
    !i  print detail tutor docs"
    !H  toggle hideinfo option"
    !k  toggle exit_sync option"
    !l  print host list"
    !L  peek the log file"
    !n  specify and repeat a command"
    !o  toggle login_only option"
    !p  toggle persistent option"
    !P  go to tclsh"
    !q  toggle nointeract option"
    !r  repeat the previous cmds1 cmds execution"
    !s  stop the remaining of previous automations"
    !t  toggle timestamp option"
    !T  toggle timestamp option"
    !v  print version/license info"
Ctrl-g suspend the script"
Ctrl-\(SIGQUIT) to stop the cmds1 automations"

{master}
lab@alecto-re0>

7. license

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

8. appendix

telnet client settings

as an expect/tcl script, crtc just call external client tools to login remote device. but there might be some subtle difference between host to host in terms of the default settings of these clients software, and this might cause some potential issues to the login automation.

for example if the telnet protocol was configured to login, in some jtac server you might notice this:

[pings@svl-jtac-tool02 ~]$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp1)

Password:

while typing the exact same commands from another server print generate different interactions:

ping@ubuntu1404:~$ telnet alecto.jtac-east.jnpr.net
Trying 172.19.161.100...
Connected to alecto.jtac-east.jnpr.net.
Escape character is '^]'.

alecto-re0 (ttyp0)

login:

this makes the same login steps configured for same remote machine works from one server, but may fail from the other.

set login_info(myrouter)       [list               \
    "$"        "telnet alecto.$domain_suffix"    \
    "login: "      "lab"                        \#<------expect a "login:" string
    "Password:"    "lab123"                   \
    ">"            "set cli screen-width 300"   \
    ">"            "set cli timestamp"          \
]

checking the telnet client default settings in both server reveals the cause:

svl server                                                          my server
========================================================================================================================
telnet> display                                                     telnet> display
will flush output when sending interrupt characters.                will flush output when sending interrupt characters.
won't send interrupt characters in urgent mode.                     won't send interrupt characters in urgent mode.
will send login name and/or authentication information.             won't read the telnetrc files.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
won't skip reading of ~/.telnetrc file.                             won't map carriage return on output.
won't map carriage return on output.                                will recognize certain control characters.
will recognize certain control characters.                          won't turn on socket level debugging.
won't turn on socket level debugging.                               won't print hexadecimal representation of network traffic.
won't print hexadecimal representation of network traffic.          won't print user readable output for "netdata".
won't print user readable output for "netdata".                     won't show option processing.
won't show option processing.                                       won't print hexadecimal representation of terminal traffic.
won't print hexadecimal representation of terminal traffic.

echo            [^E]                                                echo            [^E]
escape          [^]]                                                escape          [^]]
rlogin          [off]                                               rlogin          [off]
tracefile       "(standard output)"                                 tracefile       "(standard output)"
flushoutput     [^O]                                                flushoutput     [^O]
interrupt       [^C]                                                interrupt       [^C]
quit            [^\]                                                quit            [^\]
eof             [^D]                                                eof             [^D]
erase           [^?]                                                erase           [^?]
kill            [^U]                                                kill            [^U]
lnext           [^V]                                                lnext           [^V]
susp            [^Z]                                                susp            [^Z]
reprint         [^R]                                                reprint         [^R]
worderase       [^W]                                                worderase       [^W]
start           [^Q]                                                start           [^Q]
stop            [^S]                                                stop            [^S]
forw1           [off]                                               forw1           [\377]
forw2           [off]                                               forw2           [\377]
ayt             [^T]                                                ayt             [^T]
                                                                    telnet>

the reason is that this svl server by default have the telnet autologin feature enabled, and what that does is to send user-id of current user as the telnet login name right after the initial telnet negotiation. this ends up with only the password prompt given by the telnet server. man telnet tells more detail crabs about this behavior:

autologin     If the remote side supports the TELNET
              AUTHENTICATION option telnet attempts to use it
              to perform automatic authentication.  If the
              AUTHENTICATION option is not supported, the
              user's login name are propagated through the
              TELNET ENVIRON option.  This command is the same
              as specifying -a option on the open command.

crtc simply avoid this issue by always turning off the auto-login feature (telnet -K), if the telnet was configured in the login_info array.

8.1. add an option

adding a new option or command line flag takes 3 steps:

• add a default value in config_default

  set config_default {
      ......
      set options(verbose)                1
      ......
  }

• update optlist and optmap

  set optlist "-(a|A|b|B|c|C|d|D|e|E|f|F|g        \
                |h|H|i|I|j|J|k|K|l|L|m|M|n|N      \
                |o|O|p|P|q|Q|r|R|s|S|t|T|u|U      \
                |V|v|w|W|x|X|y|Y|z|Z)"
                   ^
                   |
                   |

  array set optmap {                  \
      ......                          \
      "-v" verbose                    \
      ......                          \
  }

• update usage

  proc usage {} {
      ......
      send_user "     -v  :verbose\n"
      ......
  }

8.2. update notes and todo list

updates:

• (2014-12-06) updates:

  • merge everything in one file: code, docs, config template.

  • config file is now optional for the script.

  • -H, !i to display this tutor

• (2014-12-7) removed sensitive info and pushed to github

• (2014-12-12) updates:

  • auto_paging mode works, no need "show …​|no-more" when turned on

• install signal intercepters: to stop expect when needed

• to support auto detection to the user-defined "issues"

todo:

• to provide a menu, easier hosts selection

• to write a unix man page

• add kibitz mode

• add automatic config scripts generation

8.3. some interesting references:

• utility conventions

• GNU standard

• ascii control codes

8.4. test set (for testing crtc dev only)

crtc -c "show chassis alarms" -H -q -n 30 -i 2 alecto@jtac
crtc -Hqi2c "show chassis alarms" -n 30 alecto@jtac
crtc -b "configure" \
crtc -E ">" -S "start shell" -E "%" -S "su" -E "sword" -S "Juniper" \
crtc -b ">" -b "start shell" -b "%" -b "su" -b "Password:" -b "lab123" \
crtc alecto@jtac
crtc -n 3 -i 5 alecto@jtac
crtc -c "show system uptime" -c "SLEEP 15"        \
crtc -c "GRES" -Hqn 30 -i 300 alecto@jtac
ver=`~pings/bin/crtc -Hqc "show version | no-more" alecto@jtac | grep -i "base os boot" | awk '{print \$5}'`
ver=`~pings/bin/crtc -Hqc "show version | no-more" $1 | grep -i "base os boot" | awk '{print \$5}'`
for i in {2..254}; do crtc -Hqac "show chassis hardware" -w 5 172.19.161.$i@jtac; done;
crtc -Ht e320-svl

crtc -yn 10 -i 20 \
  -E "$" -S "telnet alecto.jtac-east.jnpr.net" -E "login: " -S "lab" \
  -E "Password:" -S "lab123" -E ">" \
  -c "show interfaces ge-1/3/0 extensive | match pps" \
  -c "show vpls connections instance 13979:333601" \
  -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps" \
  -R "2@@2601\s+rmt\s+LD" \
  -I "1@pps!=pps_prev" \
  -Y "show interfaces ge-1/3/0.601 extensive | no-more" \
  -N "configure" -N "run show system uptime" -N "exit" \
  a_new_router

crtc -yn 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
    -I "1@pps!=pps_prev" anewrouter

crtc -n 10 -i 20 -R "2@@2601\s+rmt\s+LD" anewrouter

crtc -n 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
crtc -yn 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
crtc -n 10 -i 20 -R "2@@2601\s+rmt\s+LD" anewrouter

screen -fn -t alecto ~pings/bin/crtc alecto@jtac
dislocate crtc -H alecto@jtac
 1  29571  Sat Apr 11 11:11:28  crtc -H alecto@jtac
 2  27225  Sat Apr 11 10:29:56  crtc rams@jtac

crtc -c "show chassis alarms" -H -q -n 30 -i 10 alecto@jtac
crtc -h cenos-vm1 centos-vm2 centos-vm3
crtc -h cenos-vm{1..3}
crtc -n3i5Pc "show system uptime" alecto@jtac tintin@jtac
crtc -pr3tiUn 10000 -b "start shell" sonata@jtac
crtc -pc "show system uptime" -n 20 -i 5 -r 5 alecto@jtac
crtc -d3n 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
crtc -E "$" -S "telnet alecto.jtac-east.jnpr.net" -E
crtc -n 10 -i 20 -R "1@@Input  packets:\s+(\d+)\s+(\d+) pps@packets@pps"
crtc -n 10 -i 20 -R "2@2601\s+rmt\s+LD" anewrouter

8.5. TODO and DONE

8.6. known issue

configparser is a evil - it is convenient/quite way to dump config file content into options_cfg array for later reference, but the implementation is buggy and may cause a lot of trouble/concerns later.

with this in config: the login_jtac_server shows "[list" in options_cfg

#this implementation has some potential issue...TODO/buggy
#considering this:
#   set login_jtac_server [list             \
#           "$"     "$go_jtac_server"       \
#           "sword" "$unixpass"             \
#   ]
#

with this, SKIP_retry always shows 3 in options_cfg!

set reconnect_eval {
    ......
    global SKIP_retry1
    if ![info exists SKIP_retry1] {
        set attlab_account $attlab_account2
        set attlab_pass $attlab_pass2
        set SKIP_retry1 1
        puts "retry $SKIP_retry1 time!"
    } elseif {$SKIP_retry1==1} {
        set attlab_account $attlab_account3
        set attlab_pass $attlab_pass3
        set SKIP_retry1 2
        puts "retry $SKIP_retry1 time!"
    } elseif {$SKIP_retry1==2} {
        set attlab_account $attlab_account4
        set attlab_pass $attlab_pass4
        set SKIP_retry1 3
        puts "retry $SKIP_retry1 time!"
    } else {
        set attlab_account $attlab_account5
        set attlab_pass $attlab_pass5
        puts "SKIP_RETRY1 is $SKIP_retry1........................"
        unset SKIP_retry1
        puts "too much wrong login, will exit..."
    }
    ......
}

currently workaround is to not dump options_cfg to opt, and options_cfg to options,if option name looks "SKIP_".

tcl notes

9. tcl man pages

install tcl-doc package:

sudo apt-get install tcl-doc

then:

man lappend

for "overlapping man pages" - command that is same as other unix tools, e.g. lsearch:

ping@ubuntu47-3:~$ whatis lsearch
lsearch (3)          - linear search of an array
lsearch (3tcl)       - See if a list contains a particular element

then to view tcl’s lsearch man page:

man 3tcl lsearch

10. tcl var

As a matter of fact, every string is a legal Tcl variable name.

set -> abc
set 1 abc
......

10.1. ""

• just serve to group strings

• quotes themselves are not part of the parameter

10.2. {}

• braces themselves are not part of the parameter

• to "group", AND to "defer" the evaluation

• used in "control structures" (while, for, foreach..) to defer the evaluation of vars in the body or conditions - very important/fundamental concept in TCL!

example:

while {$retry_count <= $retry_max} {
}

what happened inside while:

when "while" see the condition $retry_count ⇐ $retry_max, since it is inside of braces {}, the evaluation of vars $retry_count and $retry_max will be "defered". So "while" still "see" literal $retry_count ⇐ $retry_max, instead of 0⇐4. otherwise 0⇐4 will be true forever and while won’t have chance to stop.

and, since evaluation in braces got defered, while, and some other control structures, will evaluate by itself.

same applies to for. foreach.

example:

#this won't work!:
#while [regexp {%(\d)} $string_new -> num] {}
#this works..
while {[regexp {%(\d)} $string_new -> num]} { }

if

the above process may not apply to if.

if structure just evaluate the condition once. so the condition does not need to be deferred in if ⇒ {} is not needed in that sense. but grouping is still needed in some cases.

if in the cases grouping is also not needed, {} can be omited!

if $a {puts abc}

11. string

Everything is a string in Tcl, (weird? or cool?) but functions that expect a number (like expr) will use that 'string' as an integer:

% set str " 123 "
123
% set num [expr $str*2]
246

11.1. string replace

• from original string $init_template,

• search for a string cmds1 , and

• replace it with value of var $cmds,

• place the changed string to a var (name from evaluation of init_$cmds)

  regsub -all {cmds1} $init_template "$cmds" init_$cmds

  set value [string replace $value 0 0]

  regsub -all \
      {\$regex4onecmd} $code_template \
      "\{$regex4onecmd\}" temp

  regsub -all $word $issue4onecmd "\$$word" issue4onecmd
  regsub -all {\$issue4onecmd_holder} $temp "\{$issue4onecmd\}" temp

regsub with backreference:

regsub -- {([^\.]*)\.c} file.c {cc -c & -o \1.o} ccCmd

scan $cmd_output, locate all braces and escape them:

if {[regsub -all {([{}])} $cmd_output {\\\1} cmd_output] > 0} {
    myputs "substituted cmd_output now looks $cmd_output" 3
}

eval [subst {regexp {$regex} {$cmd_output} -> $vars}]]

this is sometime very useful and a must, otherwise if cmd_output contains unmatched braces, the statements to be eval.ed will bail out with errors.

11.2. string repeat

myputs2 "\n\n[string repeat "- " 10]"
myputs2 "\n[string repeat "- " 20]\n"
regsub {SSH} $value "ssh[string repeat " " 5]" newvalue

11.3. string trim

to remove a matching "substring"

send -i $session [subst [string trim $anti_idle_string '"']]

11.4. string index

set firstchar [string index $value 0]

11.5. an empty string

its hard to implement an "empty" string sometime..

these don’t work:

crtc -a 'set prefix_mark ""' -c "show version %T" myrouter
crtc -a "set prefix_mark \"\"" -c "show version %T" myrouter

the final value of prefix_mark is two literal quotes: "", not empty.

workaround is to use 0 to indicate "nothing".

crtc -a "set prefix_mark 0" -c "config" -c "save backup%H_%T" myrouter

11.6. scan

if {[scan $buf "search %s" domainname] == 1} {
}

11.7. format

very nice char - ctrl-char translation: P330

spawn $env(SHELL)
interact -re "/\\\\(.)" {
    scan $interact_out(l,string) %c i
    send [format %c [expr $i-96]]
}

12. list

12.1. lsearch

if {[lsearch -exact $vars $word]!=-1} {
set opt_ind [lsearch $p_argv2 $opt]
set ix [lsearch -exact $list $value]
set hostnum [lsearch [array names session2host] $session1]
if {[lsearch -exact $list $item] != -1} {
if {[lsearch -exact $list $item] != -1} {
if {[lsearch "pings ping" $env(USER)] < 0} {
if {[lsearch "pings ping" $env(USER)] < 0} {}
if {[lsearch "pings ping" $env(USER)] < 0} {
set hostlist_all_except_last [lrange $arglist [lsearch $arglist "-h"]+1 end]
if {[lsearch $email_on_event "EMAIL_LOG_ON_LOGIN"]>=0} {

12.2. lrange

set vars [lrange $regex_vars_list 3 end]
set arglist [lrange $argv 0 end-1]
set hostlist_all_except_last [lrange $arglist [lsearch $arglist "-h"]+1 end]
[lrange $cmd_list $repeat_pos-$repeat_num $repeat_pos-1]

12.3. llengh

set cout_llen [llength $cout_list]
if {[llength $args]} {}

12.4. lindex

set argvn [lindex $argv end] ;#get the last param

13. array

expect1.17> set aa1(1 2） 3
wrong # args: should be "set varName ?newValue?"
    while executing
"set aa1(1 2） 3"

expect1.15> set "aa1(1 2)" 3
3
expect1.16> parray aa1
aa1(1 2) = 3

14. control structures

14.1. switch

switch is functionally same as if else, but in a more compact form.

switch -exact -- $user_key {
    "i" {
        ......
    }
    "s" {
        ......
    }
    default {
        ......
    }
}

switch -exact -- $argvn {
    "-G" {              ;#{{{5}}}
    }
    "-H" {              ;#{{{5}}}
        system less "~/.crtc-tutor.txt"
    }
    "-K" {              ;#{{{5}}}
    }
    default {           ;#{{{5}}}
        puts "unsupported parameter or need a session name:\'$argvn\'!"
        usage
    }
}

foreach {dash value} $arglist {
    switch -exact -- $dash {
        #-A/-b/-B      ;#{{{5}}}
        "-A" { eval $value }
        "-b" { lappend pre_cmds_cli($data_index) $value; }
        "-B" { lappend post_cmds_cli($data_index) $value; }
        #-e/-s/-c     ;#{{{5}}}
        #"-c" { lappend cmds1($data_index) $pattern_common_prompt $value; }
        "-e" -
        "-s" -
        "-c" { lappend cmds_cli($data_index) $value; }
    }
}

Note	an annoying issue caused by braces in comment (again!):

switch -exact -- "abc" {
    "blabla" {
    }
    default {
        #} "reversely" matched braces here {
        puts "this won't be executed!"
    }
}

if you are lucky, you’ll get below kind warning:

extra switch pattern with no body

if you are not, it just ignore "default" clause, and won’t warn anything!

% switch -exact -- "abc" {
    "blabla" {
    }
    default {
        #}  send -i exp6 {show system uptime
        puts "this won't be executed!"
    }
}
%

% switch -exact -- "abc" {
    "blabla" {
    }
    default {
        #} "reversely" - matched braces here {
        puts "this won't be executed!"
    }
}
%

same error will be triggered if leaving some comments and enf of each block:

switch -exact -- $user_key {
    "q" {   ;#{{{4}}}
        #close; wait
    }

    default {       ;#{{{4}}}
        if $select_host {
        } else {
        }   ;#else
    }       ;#default
}           ;#switch

an example

this is wrong: trying to use a empty string, will always get a match. in below code the "default" clause will never be examined.

switch -regexp -- $action_name {
    "RECONNECT.*" {   ;#{{{5}}}
    ......
    }
    "EXIT.*" {   ;#{{{5}}}
        exit
    }
    "RETRY.*" {   ;#{{{5}}}
    }
    "CTRLC.*" {   ;#{{{5}}}
    }
    "CONTINUE.*" {   ;#{{{5}}}
    }
    "" {   ;#{{{5}}}
        send_user -- "no action name configured!"
    }
    default {       ;#{{{5}}}
        send_user -- "<<<will execute configured action\
            group -$action_name-!!"
        puts "\n\\\"$action_name\\\" looks a normal cmdgroup"
        puts "will execute"
        exec_cmds $router $action_name
    }
}

14.2. -regexp -matchvar

http://www.tcl.tk/man/tcl8.5/TclCmd/switch.htm

this is handy:

set bar "aa 11 22 bb abc def"
switch -regexp -matchvar foo -- $bar {
    a(b*)c {
        puts "matched var food is -$foo-"
        puts "Found [string length [lindex $foo 1]] 'b's"
    }
    d(e*)f(g*)h {
        puts "matched var food is -$foo-"
        puts "Found [string length [lindex $foo 1]] 'e's and\
            [string length [lindex $foo 2]] 'g's"
    }
}

matched string will be saved in a list named "foo", in the way similiar to expect_out:

• first elments is the full matched string

• 2nd element is the info captured in the first parenthesis

• 3nd element is the info captured in the 2nd parenthesis

• so on so forth

result:

matched var food is -abc b-
Found 1 'b's

15. array operation

• to backup/copy an array:

  array set regex_info_backup [array get regex_info]

• to assign value

  array set optmap {                   \
      "-a" attribute                   \
      "-A" auto_paging                 \
  }

16. string vs. list

not everything is a list

list requires some certain format.

tclsh> set y3 { a b "My name is "Goofy"" }
a b "My name is "Goofy""

tclsh> lindex $y3 2
list element in quotes followed by "Goofy""" instead of space

There is nothing wrong with y3 as a string. However, it is not a list.

things are diff when looking as a string and list

set y {a b {Hello world!}}
set z {a [ { 1 } }

% set y {a b {Hello world!}}
a b {Hello world!}
% string length $y
18
% llength $y
3

string y, has 18 chars as a string, and 3 elements as a list

when list processing doesn’t work easily, try string processing

>set reconnect_eval {
    set login_info(myrouter) [list                  \
        "$" "telnet alecto-re0."      \
        "login: " "b"                  \
        "Password:" "b"                 \
        ">" "set cli screen-width 300"              \
        ">" "set cli timestamp"                     \
    ]
}

>set b [list "ogin" $reconnect_eval]

ogin {
set login_info(myrouter) [list                   "$" "telnet alecto-re0."       "login: " "b"                   "Passw
ord:" "b"                  ">" "set cli screen-width 300"               ">" "set cli timestamp"                      ]
}

>set c [lrange $b 1 end]
{
set login_info(myrouter) [list                   "$" "telnet alecto-re0."       "login: " "b"                   "Passw
ord:" "b"                  ">" "set cli screen-width 300"               ">" "set cli timestamp"                      ]
}

eval {$c}
invalid command name "{
set login_info(myrouter) [list                   "$" "telnet alecto-re0."       "login: " "b"                   "Passw
ord:" "b"                  ">" "set cli screen-width 300"               ">" "set cli timestamp"                      ]
}"
while evaluating {eval {$c}}

>set d [string trimright [string trimleft $c \{] \}]

   set login_info(myrouter) [list                   "$" "telnet alecto-re0."       "login: " "b"                   "Pa
ssword:" "b"                  ">" "set cli screen-width 300"               ">" "set cli timestamp"
  ]
>eval $d
{$} {telnet alecto-re0.} {login: } b Password: b > {set cli screen-width 300} > {set cli timestamp}

16.1. split and join: string vs list

• split: split a string to a list

• join: join a list into a string

split

set cout_list_prev [split $cmd_output_prev "\n"]
set regex_vars_list [split $regex4onecmd "@"]

tcl split and join is tricky: you split a string into a list then join them back, you got a different string!

e.g, this won’t work sometime

set regex_info_resolve($login_index) \
    [join [list "1" $line_num $regex $the_vars] "@"]

this works better:

set regex_info_resolve($login_index) \
    [list [join [list "1" $line_num $regex $the_vars] "@"]]

TODO

16.2. append vs. lappend

roughly,

• append is string operation, and lappend is list operation

• append won’t leave a space between the two elments, lappend will.

• append essential

  set var "var$string"

  |
  |
  v

  append var "abc" "def"

• lappend essential

  set list "$list $newlist"

  |
  |
  v

  lappend list $newlist

• more examples

  set line_new [append line_new "\n"]
  append buf [set expect_out(buffer)]
  append session_input_patterns [subst {
  append handler_proc "global init_$cmds\n"

  lappend list $array($name)
  lappend event_action_list "$user_pattern" "$user_action"

17. list , concat ,eval

list: preserve the original list (don’t disassamble)

tclsh8.6 [/opt/ActiveTcl-8.6/bin]concat a b "hello world"
a b hello world
tclsh8.6 [/opt/ActiveTcl-8.6/bin]list a b "hello world"
a b {hello world}

concat

group all elements of all lists into a new list - disassamble first level sub-list and put them together as a new list.

tclsh8.6 [~]concat a b "Hello world"
a b Hello world

% concat a b {c d e} {f {g h}}
a b c d e f {g h}

eval

• process parameters exactly the same way as concat:

• treat all its arguments as list

  • The elements from all of the lists are used to form a new list that is interpreted as a command.

  • The first element becomes the command name.

  • The remaining elements become the arguments to the command.

• do $ and command substitution

• use list in arguments if you don’t want them to be broken up

  compare:

  tclsh8.6 [~]eval append v2 {a b } {c {d e}}
  abcd e
  tclsh8.6 [~]eval append v3 [list {a b}] [list {c {d e}}]
  a bc {d e}

Eval takes one or more arguments, which together comprise a Tcl script containing one or more commands. Eval concatenates all its arguments in the same fashion as the concat command, passes the concatenated string to the Tcl interpreter recursively, and returns the result of that evaluation (or any error generated by it). Note that the list command quotes sequences of words in such a way that they are not further expanded by the eval command.

in Expect book, one example:

this is wrong:

spawn [lrange $argv 1 end] ;# WRONG!

this is correct:

eval spawn [lrange $argv 1 end]

reason:

spawn [lrange $argv 1 end] =>
spawn "sleep 5" =>
"sleep 5" is ONE string as a whole, but there is no such unix command

eval spawn [lrange $argv 1 end] =>
eval spawn "sleep 5" =>
concat {spawn "sleep 5"} and execute the code =>
execute code: "spawn sleep 5"

18. backslash

P92: Just remember two rules: 1. Tel translates backslash sequences. 2. The pattern matcher treats backslashed characters as literals. These rules are executed in order and only once per command.

19. exec vs. system

exec:

fork a unix process and execute it as a child-process

system:

similiar to fork, but:

• no I/O redirection

• internally much faster

• process parameters like concat (like eval + exec)

this works:

append content "\nrunning expect: [exp_version] @ [exec which expect]"
puts "$content"

result:

running expect: 5.45 @ /usr/bin/expect

this doesn’t (system does not redirect its output)

append content "\nrunning expect: [exp_version] @ [system which expect]"
puts "$content"

result:

/usr/bin/expect
running expect: 5.45 @

20. subst flags

% set abc {
 -nocase -re {\(no\)} {
     myputs "detected -\(no\)-"
     myputs "will send cmd -yes-, and continue expect"
     send -i [set session] {yes\r}
     mysleep 30
     exp_continue
 }
}

-nocase -re {\(no\)} {
    myputs "detected -\(no\)-"
    myputs "will send cmd -yes-, and continue expect"
    send -i [set session] {yes\r}
    mysleep 30
    exp_continue
}

% set session 123
123
% subst $abc

 -nocase -re {(no)} {
     myputs "detected -(no)-"
     myputs "will send cmd -yes-, and continue expect"
}    send -i 123 {yes
     mysleep 30
     exp_continue
 }

% subst -nobackslashes $abc

-nocase -re {\(no\)} {
    myputs "detected -\(no\)-"
    myputs "will send cmd -yes-, and continue expect"
    send -i 123 {yes\r}
    mysleep 30
    exp_continue
}

% subst -nobackslashes -nocommands $abc

-nocase -re {\(no\)} {
    myputs "detected -\(no\)-"
    myputs "will send cmd -yes-, and continue expect"
    send -i [set session] {yes\r}
    mysleep 30
    exp_continue
}

%

20.1. example

good one:subs backslashes {{{1}}}                                              1 bad one: no subs backslashes {{{1}}}
[Thu Apr 21 14:31:37 EDT 2016]:[myrouter]:spawn_login:..substituted      2 [Thu Apr 21 14:27:36 EDT 2016]:[myrouter]:spawn_login:..substit
    ;#{{{3}}}                                                            3      ;#{{{3}}}
+--  6 lines: set oldtimeout 200000                                  +   4 +--  6 lines: set oldtimeout 200000
                myputs2 "session:[myrouter]:you typed ESC key here..    10                 myputs2 "session:\[myrouter\]:you typed ESC key
                myputs2 "                                               11                 myputs2 "\nyou have the control now...\n"
you have the control now...                                          +  12 +-- 28 lines: mycatch "stty -raw"
"                                                                       40                                     send_user "==output for  my
+-- 28 lines: mycatch "stty -raw"                                    +  41 +-- 97 lines: ;
                                    send_user "==output for  myroute   138                                     send_user "\n[subst [clock
"                                                                    + 139 +--  3 lines: } else {
+-- 97 lines: ;                                                        142                                 exp_send -i exp6 {~8N\vV.b\r} (1)
                                    send_user "                      + 143 +--165 lines: return "RETURN_EXPECT_SENDFIRST0_NORMAL"
[subst [clock format [clock seconds] -format $dateformat]](local)
"
+--  3 lines: } else {
}                               exp_send -i exp6 {~8N\vV.b      (2)
+--165 lines: return "RETURN_EXPECT_SENDFIRST0_NORMAL"
[Thu Apr 21 14:31:37 EDT 2016]:[myrouter]:spawn_login:..get new stri   ~

1. working example: "\r" got substituded - hence removed from the password char

2. non-working example: "\r" retained because of subst -nobackslashes knob, and so \r will be treated as part of password chars. this will run into issues.

21. dynamic coding: eval + subst

21.1. example1

we have array cmds2 defined, along with other arrays with similiar name cmdsN:

set cmds2(myrouter) [list   \
    "show chassis alarm"    \
    "show system uptime"    \
]

set cmds3(myrouter) [list   \

set cmds4(myrouter) [list   \

set cmds5(myrouter) [list   \

...

now we want to refer the element of one of the array, but in a dynamic way that, to which array we are referring to need to be a variable that can be changed as needed in the run time, for that purpose you may create code like this:

this doen’t work:

foreach a_cmd $cmds($login_index)] {
    puts "get a cmd $a_cmd"
}

cmds(myrouter) looks just like a new array, which is not defined, and also not what we wanted

this doesn’t work:

foreach a_cmd ${cmds}($login_index)] {
    puts "get a cmd $a_cmd"
}

with {} the cmds will not associate with (myrouter) now, but:

1. ${cmds} evaluate to cmds2

2. we now get a wrong foreach statement:

   foreach a_cmd cmds2(myrouter)

here we are expecting to iterate in a "value" $cmds2(myrouter), not a variable name cmds2(myrouter).

these doesn’t work

foreach a_cmd $[set cmds]($login_index)] {
    puts "get a cmd $a_cmd"
}

foreach a_cmd $${cmds}($login_index)] {
    puts "get a cmd $a_cmd"
}

These seems to be right, but we’ll end up with a literal string $cmds2(myrouter), not the value of variable cmds2(myrouter)

get a cmd $cmds2(myrouter)

this works:

set cmds cmds2
foreach a_cmd [set ${cmds}($login_index)] {
    puts "get a cmd $a_cmd"
}

what it does:

1. use ${cmds}() to make it not look like an array cmds(myrouter)

2. now we got [set cmds2(myrouter)]

3. the [] will evaluate set cmds2(myrouter) expression, resulting in the value of variable cmds2(myrouter)

another way is to use subst to evaluate the value of the array

set cmds cmds2
foreach a_cmd [subst $[set cmds]($login_index)] {
    puts "get a cmd $a_cmd"
}

21.2. example2

this is wrong:

set i 101
set cmds${i}($login_index) $a_cmd_resolved
myputs "get an array cmds$i: $cmds$i($login_index)"

the goal is to say:

get an array cmds101: "show system uptime"

but the $cmds look like a variable, which does not exist, and not what we wanted. we need to:

1. make cmds to group with $i and get cmds101

2. get the value of cmds101(myrouer): $cmds101(myrouter)

this workaround works:

myputs "get an array cmds$i: [set [subst cmds$i]($login_index)]"

21.3. cautions with subst

considering this dynamic code:

set pattern "(% |> |# |\\\$ |%|>|#|\\\$)$"

set myexpectcmd {
    expect {
        ......
        -re "$pattern" {
            myputs "expected pattern -$pattern_slash- captured"
            ......
        }
        ......
    }
}

puts "myexpectcmd looks:\n[subst $myexpectcmd]"
eval [subst $myexpectcmd]

the original pattern will look:

-re "(% |> |# |\\\$ |%|>|#|\\\$)$" {...}

subst will evaluate a regex one more time, so after subst, the orignial expect statement -re "$pattern" will become:

-re "(% |> |# |\$ |%|>|#|\$)$" {...}

and this will cause issues in the original expect code.

to solve this there are at least 2 solutions:

• to "compensate" some lost \

• protect with {}

this works

if ![regsub -all {\\\$} $pattern {\\\\\$} pattern_slash] {
    set pattern_slash $pattern
}

set myexpectcmd {
    expect {
        ......
        -re "$pattern_slash" {
            myputs "expected pattern -$pattern_slash- captured"
            ......
        }
        ......
    }
}

so the original pattern now will appear the same when eval see it. to illustrate the change:

% set pattern "(% |> |# |\\\$ |%|>|#|\\\$)$"
(% |> |# |\$ |%|>|#|\$)$
% regsub -all {\\\$} $pattern {\\\\\$} pattern_slash
2
% set pattern_slash
(% |> |# |\\\$ |%|>|#|\\\$)$

this works

set pattern "(% |> |# |\\\$ |%|>|#|\\\$)$"

or (not tested):

set pattern {(% |> |# |\\$ |%|>|#|\\$)$}

and then:

set myexpectcmd {
    expect {
        ......
        -re {$pattern} {
            myputs "expected pattern -$pattern_slash- captured"
            ......
        }
        ......
    }
}

21.4. cautions with eval

eval is always tricky, although it’s powerful.

set expectcmd {
    #puts "user_action is $user_action"
    send_user "$expect_out(buffer)\n"
    puts "user_action configured as \"RETRY\""
}

eval [subst $myexpectcmd]

there are at least 3 issues in above code:

• the subst will resolve all var-looking strings, even in comment

• the resolution may not work - it may contains unmatched braces that broke the whole statement expectcmd

• the \ won’t work as expected.

quick solutions/workarounds:

• avoid having var in comments

• use one more nested var substitution to avoid being resolved.

• avoid using \

below codes work better:

set send_user_expect_out {
    catch {send_user "$expect_out(buffer)\n"}
}

set expectcmd {
    $send_user_expect_out
    puts "user_action configured as RETRY"
}

eval [subst $myexpectcmd]

22. continuation

1. explicit continuation: a back slash at the end: \ will be replaced by one space

2. implicit continuation: an open brace at the end

23. tcl comment

Tcl comments behave a lot like commands. They can only be used where commands can be used

comment can’t include unmatched braces

this is wrong:

#this is a comment {
set a "abc"

and this is wrong:

#this is a comment }
set a "abc"

commment can’t be appended behind interact

this is wrong:

interact { #this is our interact
    ......
}

this is correct:

#this is our interact
interact {
    ......
}

even if the number of braces matched, but still this will cause issue:

if 1 {
#myputs "will send cmd -show system uptime-, "
#}  send -i exp6 {show system uptime        #<------
#mysleep 30
puts "this won't be executed"
}

this is the error:

wrong # args: extra words after "else" clause in "if" command

Note	this will cause the "default" clause in switch command being dropped "silently"!

24. tcl error processing: catch

the simplest form

put tcl statement in a brace and catch, this will prevent the script from exit on error.

catch {
    #send_user "$expect_out(buffer)\n"
    #myputs "output looks -$expect_out(buffer)-"
    send_user "$output_new\n"
}

if [catch "spawn -noecho $env(SHELL)" reason] {
    ......
}

catch {exec "grep $login_index $config_file"}

wrap with a proc:

proc mycatch {cmd} {    ;#{{{2}}}
    if { [catch {eval $cmd} msg] } {
       puts "Something seems to have gone wrong:"
       puts "Information about it: $::errorInfo"
       return 1
    } else {
        return 0
    }
}

mycatch "stty -raw"

25. file operation

25.1. file command

set h_debugfile [open $debugfile w]
set file [open $file r]

tclsh8.6 [~]set abc /home/ping/bin/crtc/crtc.conf
/home/ping/bin/crtc/crtc.conf

rootname

tclsh8.6 [~]file rootname $abc
/home/ping/bin/crtc/crtc

extension

tclsh8.6 [~]file extension $abc
.conf

dirname

tclsh8.6 [~]file dirname $abc
/home/ping/bin/crtc

tail

tclsh8.6 [~]file tail $abc
crtc.conf

basename

tclsh8.6 [~]exec basename $abc
crtc.conf

tclsh8.6 [~]file rootname [file tail $abc]
crtc

tclsh8.6 [~]file executable $abc
0

25.2. open/read/flush/fconfigure

open

this is a very typical file open/read/close process…​

if [file exists $file] {
    set file [open $file r]
    while {[gets $file buf] != -1} {
        if {[scan $buf "search %s" domainname] == 1} {
            close $file
            return $domainname
        }
    }
    close $file
    error "no domain declaration in $file"
    return 0
} else {
    puts "no file $file exists!"
    return 0
}

another commonly used technique is to wrap open with catch

if [catch {open $infile r} fp] {
    #sth wrong
    return 1
} else {
    #use $fp to read ...
}

flush

force buffered data out

gets

read a line at a time, and return length of strings read, or -1 when done

while {[gets $file line] != -1} {
    # do something with $line
}

read

read a fix number of characters, or entire file if not specified.

set chunk [read $file 100000]

eof

return 0 when eof encountered

read 100k chars at a time, until end of file.

while {![eof $file]} {
    set buffer [read $file 100000]
    # do something with $buffer
}

read whole file in a buffer, then process each line

foreach line [split [read $file] "\n"] {
    # do something with $line
}

25.3. pipeline: work with existing unix command!

tclsh8.6 [~/temp]echo "a1 100" > a1.txt
tclsh8.6 [~/temp]echo "b1 200" >> a1.txt
tclsh8.6 [~/temp]echo "a2 90" >> a1.txt
tclsh8.6 [~/temp]cat a1.txt
a1 100
b1 200
a2 90
tclsh8.6 [~/temp]set input [open "|sort a1.txt" r]
file6
tclsh8.6 [~/temp]read $input
a1 100
a2 90
b1 200

25.4. cd, pwd

26. proc

P244, good sum of info sharing method between caller and proc:

1. global

2. return value

3. upvar a a to bind var a in caller to var a in proc

4. upvar $a b to "pass as reference"

optional parameter

proc myputs {msg {level 1} args} {
}

26.1. upvar: passing by reference (pointer)

proc myproc {name1 name2} {
    upvar $name1 p1 $name2 p2
    set p1 1
    set p2 2
}

set n1 10
set n2 20
myproc n1 n2        #<------

puts "n1,n2 now changed to $n1,$2"

result is:

n1,n2 now changed to 1,2

Note	it has to be myproc n1 n2 - call by name. not myproc $n1 $n2 - call by value

                    VVVVVVVVVV VVVVVVVVVVVVVVVV
proc do_pag {router cmds_array cmd_output_array {pa_intv 0} {pattern_timeout 120} {pa_pair 1}} {

    upvar $cmds_array p_cmds_array
    upvar $cmd_output_array p_cmd_output_array
    ...
    set p_cmds_array(x) y
}

set cmds "cmds1"
do_pag $login_index $cmds cmd_output_array_${cmds}_prev $interval_cmd $waittime_cmd $pa_pair]
                    ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

this equals to:

do_pag $login_index cmds1 cmd_output_array_cmds1_prev $interval_cmd $waittime_cmd $pa_pair]
                    ^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    passing name, not value, to proc

26.2. upvar vs global

sometime I :

• need a variable to pass value across function calls

• want to avoid to use function parameters.

• avoid using global, because the variable is not important in the rest part of the code. However,

  proc myinteract {router} {
      ......
      set command ""
      ......
  }

  proc interact_c {router} {

      ......
      upvar command command1
      upvar router router1
      upvar commandgroup commandgroup1
      global global_data; eval $global_data
      ......
  }

per P244: the two vars can have same name, so simply:

set var 5
proc setvar {} {
    upvar var var
    set var 10
}
setvar
puts "$var"

this will print 10, not 5.

in upvar var var, first var belongs to caller’s scope, while 2nd belongs to process scope.

26.3. upvar to associate var1 var to var2 name

suppose:

>set a "b"
b
>set b "bvalue"
bvalue

print value of c:

>set c [set $a]
bvalue

same can be done:

>upvar 0 $a x
>set c $x
bvalue

>set y $a
b
>set c $y
b

26.4. uplevel

proc exec_cmds2 {login_index {cmds "cmds1"}} {
    uplevel {eval $update_code_template}
}

proc interact_connection_close {} {
    uplevel {set is_respawn 1}
}

to build a customized "return":

proc myreturn {args} {
    if {[llength $args]==0} {uplevel return}
    if {[llength $args]==1} {uplevel [list return [lindex $args 0]]}
    if {[llength $args]==2} {
        set funcname [lindex $args 1]
        puts "==> leaving $funcname now"
        uplevel [list return [lindex $args 0]]
    }
}

the list is necessary, otherwise below code won’t work:

proc myreturn {args} {
    ...
    if {[llength $args]==1} {uplevel {return [lindex $args 0]}}
    ...
}

unfortunately this still doesn’t work well. below proc proc1 will actually return 2, not 1.

proc proc1 {} {
    myreturn 1 "proc1"
    return 2
}

26.5. unknown

tclsh8.6 [~]proc unknown args {
>puts "abc"
>}
tclsh8.6 [~]aaa
abc

tclsh8.6 [~]aaaa
abc

27. argv

example1: $argv/$argc/$argv0/$argv N

             count is $argc
             -------------list is $argv
myscript.tcl param1 param2
------------ -----  ------
$argv0       |      |
             |      |[lindex argv 1]
             |[lindex argv 0]

• '$argv0' as script name,

• '$argv' as a list of parameters following script name

• '$argc' is size of list

• '[lindex $argv 0]' as 1st param,'[lindex $argv 1]',etc

28. namespace

seesm no much usage for me.

29. regexp and regsub

capture all match, and return a list

set issue_words [regexp -all -inline {(\w+)} $issue4onecmd]

capture all "submatch"?

seems no "built-in" support on this. but not hard to get, based on the result returned from above.

http://stackoverflow.com/questions/17176121/getting-values-for-all-submatches-alone-leaving-the-match-in-regexp-inline

% set ospf

Feb 20 22:52:04
Address          Interface              State     ID               Pri  Dead
10.192.0.41      xe-3/1/0.0             Full      192.168.0.6      128    35
10.192.0.45      xe-4/1/0.0             Full      192.168.0.7      128    33
1.1.1.2          so-1/2/0.0             Full      100.100.100.100  128    39
2.2.2.2          so-1/2/1.0             Full      20.20.20.20      128    34

% regexp -all -inline {(\S+)\s+Full} $ospf
{xe-3/1/0.0             Full} xe-3/1/0.0
{xe-4/1/0.0             Full} xe-4/1/0.0
{so-1/2/0.0             Full} so-1/2/0.0
{so-1/2/1.0             Full} so-1/2/1.0

regexp and resub can be used together to accomplish sth like this:

if [regexp {(ssh\s{1,4})\S} $value -> tobereplaced] {
    myputs "found pattern -$tobereplaced-" 3
    regsub "$tobereplaced" $value \
        {ssh -o "StrictHostKeyChecking no" } newvalue
    myputs "step -$value- became newstep $newvalue" 3
}

• use a pattern to match

• but just to replace the sub-pattern, not the whole pattern

seems no good way to do same in one regsub

put regexp in a variable:

eval [subst {regexp {$regex} {$cmd_output} -> $vars}]]

30. to kill a telnet/ssh session

method1: to send a logout command, e.g, in Junos it’s "exit"

send -i $session "exit\r"

method2: use expect close and wait command, to close a spawned process

catch {close $session;wait $session}

31. logical operation

if {[info exists options_cli(env_proof)] && \
         ( ($options_cli(env_proof) != 0) || \
          ![string equal $options_cli(env_proof) ""]\
         )
    } {
        puts abc
}

31.1. string compare

this works:

% if {1!=2} {puts abc}
abc

This doesn’t:

% if {[1!=2]} {puts abc}
invalid command name "1!=2"

To workaround use this:

% if {![string equal 1 2]} {puts abc}
abc

31.2. ternary operator

set myexpectcmd [expr $slow_mode ? myexpectcmd_slow : myexpectcmd]

expect1.4> set a 0
0
expect1.7> set b [expr $a?0:1]
1
expect1.8> set a 1
1
expect1.10> set b [expr $a?0:1]
0

this doesn’t work:

expect1.6> set b [$a?1:0]
invalid command name "0?1:0"
    while executing
"$a?1:0"

32. debugging

trace function call:

#in the begging
myputs "==>entering reload_data"

#in the end
myputs "==>leaving reload_data"

33. cron/event-scheduler-like

trace cmd

proc at {time args} {
  if {[llength $args]==1} {set args [lindex $args 0]}
  set dt [expr {([clock scan $time]-[clock seconds])*1000}]
  after $dt $args
} ;# RS
at 9:31 puts Hello
at 9:32 {puts "Hello again!"}

proc atNext {time fmt cmd} {
     return [after [expr {([clock scan $time -format $fmt]-[clock seconds])*1000}] $cmd]
}
# Example:
puts [atNext {Saturday 10:00} {%A %H:%M} {puts "Hallo Welt"}]
vwait forever

34. nested function call

be cautious about the return value.

35. tcl error messages

"too many nested evaluations (infinite loop?)"

problem of recursive function call.

36. coding issues

related code:

myinteract {
    ......
    if {<$enable_user_patterns || $enable_user_patterns==1} {
        set myinteract_session_input_patterns ""
    } else  {
        set myinteract_session_input_patterns2 ""       (1)
    }

    set interactcmd [subst {
        interact {
            -input $user_spawn_id
                $myinteract_user_input_patterns
            -output $session_output
            -input $session_input
                $myinteract_session_input_patterns
                $myinteract_session_input_patterns2
            -output $user_spawn_id
        }
    }]

    myputs "interactcmd looks $interactcmd" 3

    if !$nofeature {
        #eval $interactcmd {{{4}}}
        eval $interactcmd
    }
    ......
}

1. without this, there will be a VERY TRICKY issue with user_patterns

testcase:

crtc myrouter, login via jtac-server then telnet to router

issue:

generate a file named test:

echo "Connection closed by foreign host" > test

now test the persistent/RECONNECT action, defined in user_pattern:

this doesn’t work:

labroot@alecto-re0> file show test
Apr 23 22:43:19
Connection closed by foreign host

this will work:

labroot@alecto-re0> start shell
Apr 23 22:43:21
% cat test
Connection closed by foreign host
interact: detected event:
-Connection closed by foreign host-
matches pattern:
-Connection closed by foreign host-
now execute action group -RECONNECT-!!
action is RECONNECT
persistent mode set, willlll reconnect in 10s

<<<<count {10}s before proceeding...

removing the jtac server from login process, make it directly login to router.

if !$in_jserver {
    set login_info($login_index) [subst {    \
        $login_jtac_server                     \
        $login_info($login_index)            \
    }]
}

this will work.

and, exiting from router won’t trigger pattern match, but exiting from jtac server will.

labroot@alecto-re0> exit
Apr 23 22:16:30
Connection closed by foreign host.
pings@svl-jtac-tool01:~$ exit
logout
Connection to 172.17.31.80 closed
interact: detected event:
received message:
-Connection to 172.17.31.80 closed-
matches pattern:
|Connection to (d{1,3}.){3}d{1,3} closed|Connection to S+ closed|Connection reset by peer-
now execute action group -RECONNECT-!!
action is RECONNECT
persistent mode set, willlll reconnect in 10s

37. crontab

37.1. error1: env

no such variable
    (read trace on "env(USER)")
    invoked from within
"set options(log_seperator)          "
    <<<<<<<<<<<<<<<<<<< new logs since: <<<<<<<<<<<<<<<<<<<<<<<
    < [time_now] $env(USER) {1} <
    <<<<<<<<<<..."
    ("eval" body line 62)
    invoked from within
"eval $config_default"
    (file "/home/ping/bin/crtc/crtc" line 6925

37.2. error2: send_tty

send_tty: cannot send to controlling terminal in an environment when there is no controlling terminal to
send to!
    while executing
"send_tty $msg"
    invoked from within
"if !$in_shell {
            send_tty $msg
        } else {
            puts -nonewline $msg
        }"
    invoked from within
"if $verbose {
        if !$in_shell {
            send_tty $msg
        } else {
            puts -nonewline $msg
        }
    }"
    (procedure "myputs2" line 4)
    invoked from within
"myputs2 "<<<CRTC:$login_index:start to login, please wait ...\n""
    ("foreach" body line 2)
    invoked from within
"foreach login_index $hostlist_full {
    myputs2 "<<<CRTC:$login_index:start to login, please wait ...\n"
    myputs2 "<<<CRTC:$login_index:to interup..."
    (file "/home/ping/bin/crtc/crtc" line 8046)

37.3. error3: stty

http://expect.sourceforge.net/FAQ.html#q27

stty: impossible in this context
are you disconnected or in a batch, at, or cron script?stty: impossible in this context
are you disconnected or in a batch, at, or cron script?stty: ioctl(user): bad file number

    while executing
"stty -raw"
    invoked from within
"subst $myexpectcmd"
    invoked from within
"if [expr {$enable_user_patterns && [array exists user_patterns]}] {

        myputs "enable_user_patterns set ($enable_user_patterns) and user_pattern..."
    (procedure "myexpect" line 102)
    invoked from within
"myexpect $router $pattern      $datasent $pattern_timeout [expr !$pa_pair] 0    "
    invoked from within
"if {[regexp {GRES\s*(\d*)} $datasent -> interval_gres]} {
                myputs "GRES command detected!"
                if {[string equal $interval_..."
    (procedure "do_pag" line 166)
    invoked from within
"do_pag $login_index  login_info cmd_output_array_login_info  $interval_cmd $waittime_login"
    (procedure "spawn_login" line 53)
    invoked from within
"spawn_login $login_index"
    ("foreach" body line 5)
    invoked from within
"foreach login_index $hostlist_full {
    myputs2 "<<<CRTC:$login_index:start to login, please wait ...\n"
    myputs2 "<<<CRTC:$login_index:to interup..."
    (file "/home/ping/bin/crtc/crtc" line 8046)

38. misc

38.1. editor/tcl shell/etc

tkcon GUI looks good, but seems not stable - hanging after running crtc

38.2. tclreadline

The tclreadline package makes the GNU Readline library available for interactive tcl shells. This includes history expansion and file/command completion. Command completion for all tcl/tk commands is provided and commmand completers for user defined commands can be easily added. tclreadline can also be used for tcl scripts which want to use a shell like input interface. In this case the ::tclreadline::readline read command has to be called explicitly.

sudo apt-get install tclreadline

then add below in ~/.tclshrc and ~/.expect.rc:

if {$tcl_interactive} {
    package require tclreadline
    ::tclreadline::Loop
}

Tip	for me, I just like the arrow up/down, tab complete.

38.3. teacup

activetcl tool, used to install extensions.

38.4. dejagnu

38.5. make script executable but not readable ..

It is often useful to store passwords (or other private information) in Expect scripts. This is not recommended since anything that is stored on a computer is susceptible to being accessed by any‐ one. Thus, interactively prompting for passwords from a script is a smarter idea than embedding them literally. Nonetheless, sometimes such embedding is the only possibility.

Unfortunately, the UNIX file system has no direct way of creating scripts which are executable but unreadable. Systems which support setgid shell scripts may indirectly simulate this as follows:

Create the Expect script (that contains the secret data) as usual. Make its permissions be 750 (-rwxr-x---) and owned by a trusted group, i.e., a group which is allowed to read it. If necessary, create a new group for this purpose. Next, create a /bin/sh script with permissions 2751 (-rwxr-s—​x) owned by the same group as before.

The result is a script which may be executed (and read) by anyone. When invoked, it runs the Expect script.

38.6. return in source file

will make the rest part skipped

38.7. NewHeadline

this will give unexpected result:

puts "download_folder looks $download_folder!"
if [file exists $download_folder] {
    puts "$download_folder does not exists"
} elseif [catch "set download_folder [pwd]"] {
    set download_folder "/var/tmp"
} else {
    error "can't locate a valid download_folder"
    exit
}

result:

download_folder looks ~/download_folder!
/home/ping does not exists

39. reference

• re_syntax - Syntax of Tcl regular expressions

• http://www.nist.gov/el/msid/infotest/dlibes.cfm

• TCL programming cookbook

shell script to automate interaction:

only this works:

shell script to automate telnet/ftp, not working for ssh

(
    echo "labroot"
    sleep 2
    echo "lab123"
    sleep 2
    echo "show version | no-more"
    echo "show config | no-more"
    sleep 20
) | telnet -K alecto-re0.ultralab.juniper.net > showver.txt

for telnet this doesn’t work:

telnet -K alecto-re0.ultralab.juniper.net << EOF
labroot
lab123
show version | no-more
EOF

for ssh this doesn’t work:

(
    sleep 5
    echo "lab123"
    sleep 2
    echo "show version | no-more"
    echo "show config | no-more"
    sleep 20
) | ssh -t -t labroot@alecto-re0.ultralab.juniper.net

pings@svl-jtac-tool01:~/bin$ ./test.sh
Warning: Permanently added 'alecto-re0.ultralab.juniper.net' (DSA) to the list of known hosts.
labroot@alecto-re0.ultralab.juniper.net's password:

expect notes

40. expect resources

http://expect.sourceforge.net/

expect over windows: http://docs.activestate.com/activetcl/8.4/expect4win/ex_usage.html

41. expect

exp_ cmd	Functions
expect	expect message from current spawned process
expect_user	expect message from stdin
expect_tty	expect message from /dev/tty

41.1. expect syntax

expect [[-opts] pat1 body1] ... [-opts] patn [bodyn]

best style in practice:

expect {
    -i $process -re $pattern {
        myputs "expected pattern -$pattern- matched!"
        puts "will sleep for $reconnect_interval and retry"
        mysleep $reconnect_interval
        puts "retry current cmd -$datasent- now..."
        continue
    }
    timeout {
        if {$escape_count < 3} {
            incr escape_count
            puts "press ctrl-c again to escape ..."
            #just repeat sending ctrlc if got timeout
            exp_send -i $process "[CONST CTRL_C]\r"
            exp_continue
        } else {
            puts "not able to escape(not seeing expected prompt\
                -$pattern-, will exit"
            exit
        }
    }
}

41.2. expect_out

Figure 1. expect_out with nested pattern P116

[root@ftosx1 Expect]# expect
expect1.1>  expect -re "I(.*)BA"
The dog is black and I like ABBA group
expect1.2>  send $expect_out(buffer)
The dog is black and I like ABBAexpect1.3>
expect1.4>
expect1.5> send $expect_out(0,string)
I like ABBAexpect1.6>
expect1.7> send $expect_out(1,string)
 like ABexpect1.8>
expect1.9>

• internal buffer : all user input (not including typo fix editting). no way to access this buffer: "The dog is black and I like ABBA group", P74

• in expect_out(buffer) : the original entire matched string, "The dog is black and I like ABBA"

• in expect_out(0, string) : the match to the pattern, "I like ABBA"

• in expect_out(1, string) : the first parenthesized subpattern that match our pattern, "like AB"

• expect_out(spawn_id) : spawn_id of matching process P254

                       |<-pattern->|     |<-pattern->|
internal buffer : xxxxx|mm(nnnn)mmm|yyyyy|m(nnnnnnn)m|zzzzz|   (1)
                            |      ^             |   ^
                            |      (6)
                            |                    |   (8)
                            | (2)
                            |                    |(7)
                            v                    v
expect_out(buffer)  :|<-             ->|<-               ->| (3)

expect_out(0,string):      |<--------->| (4)

expect_out(1,string):         |<-->| (5)

1. expect_out buffer strings in "internal buffer"

2. when a match found, move strings matched to whole expect pattern into expect_out(buffer)

3. expect_out(buffer) contains all matched strings, plus chars that came earlier but did not match

4. expect_out(0,string) contains all matched strings

5. expect_out(1,string) contains first substring in first ()

6. internal buffer now start from the char next to the previous match

7. when a new match found, go step 2

simulating expect_out behavior with "switch"

-re ".+" {  ;#{{{5}}}
    myputs "get new strings -[set expect_out(0,string)]-"
    append buf [set expect_out(buffer)]
    myputs "this make the buf looks:\n-[set buf]-"
    switch -regexp -matchvar match -- [set buf] {
        {$pattern} {        ;#{{{6}}}
            set buf_match_loc [string first \
                [lindex [set match] 0] [set buf]]
            set buf_rm_bef_matched [string replace \
                [set buf] 0 [set buf_match_loc]-1 ]
            set buf [string trimleft \
                [set buf_rm_bef_matched] [set match]]
            myputs "buf now looks:\n-[set buf]-" 3
        }
    }
}

41.3. expect -indice

this code:

expect ">"
send "configure\r"
expect -indice "#"
puts "\nstart of expect_out -----------------"
parray expect_out
puts "end of expect_out -----------------"

generated this array:

labroot@alecto-re0> configure
Entering configuration mode

[edit]
labroot@alecto-re0#
start of expect_out -----------------
expect_out(0,end)    = 70
expect_out(0,start)  = 70
expect_out(0,string) = #
expect_out(1,end)    = 1628
expect_out(1,start)  = 1627
expect_out(1,string) = $
expect_out(2,end)    = 1627
expect_out(2,start)  = 1627
expect_out(2,string) = $
expect_out(buffer)   =  configure
Entering configuration mode

[edit]
labroot@alecto-re0#
expect_out(spawn_id) = exp6
end of expect_out -----------------

41.4. expect -re

both expect and interact support OR

anchor ^ and $:

expect anchors at the beginning of whatever input it has received without regard to line boundaries.

41.5. expect -gl

implicit -gl

expect "a*"

explicit -gl

expect -gl "a*"

explit -gl is useful to match some special patterns:

expect -gl "timeout"
expect -gl "-re"

41.6. expect *

can be used to move all old (internal) buffer into expect_out(buffer), which can be accessed.

timeout {
    expect *
    puts "expect_out(buffer) now looks -[set expect_out(buffer)]-"
    puts "timeout when looking for pattern $pattern"
}

41.7. expect (nothing)

expect

expect either "timeout" or "eof"?

expect ignores patterns for which it has no spawn ids. If the expect command has no valid spawn ids at all, it will just wait. P269.

41.8. expect timeout

these are all the same:

expect {
    {timeout} {
        puts "timeout1!!!"
    }
    timeout {
        puts "timeout2!!!"
    }
    "timeout" {
        puts "timeout3!!!"
    }
}

41.9. expect eof

matches when spawned process close connection actively - before expect timeout first.

41.10. expect pattern no-op

just match, but no-operation (p190)

expect $pattern

41.11. why "\\\$"?

set pattern "(% |> |# |\\\$ |%|>|#|\\\$)$"

in a typical tcl regex pattern usage environment, it will be scanned and processed twice:

• tcl: "\\\$" → "\$"

• regex: "\$" → literal $ sign

see P330 and P91

now, with subst and eval, one more scan will be gone through:

• subst \\\$ → \$

• tcl \$ → literal $ sign

• regex $ interpretted as "end of the string"

so if goal is to use a regex to indicate a literal $, then the original pattern needs to be either protected from being evaluated, or compenstated with more \.

more rules: P126

"non-substitution" behavior: occurs anywhere "$" is not followed by an alphanumeric character such as in the string "$+".

it looks,

• everything can be a tcl var

  tclsh8.6 [~]set + abc
  abc

• not any var will be substituted

  tclsh8.6 [~]puts $+
  $+
  tclsh8.6 [~]puts "$+"
  $+

  tclsh8.6 [~]puts "[set +]"
  abc

41.12. glob vs. regex

• ^

• $

• \ the next char literally

  foreach filename [glob *.expl {
      set file [open $filenamel
          # do something with $file
      close $file
  }

• [] a range of char

• * anything (.* in regex)

• ? single char (. in regex)

• {} a choice of string (what is this?)

• ~ home

glob vs. regex:

P126:

for regex: unless preceded by a backslash, the $ is special no matter where in the string it appears. The same holds for "^".

expect -re "% $|foo"

for glob: the glob pattern matcher only treats " as special if it appears as the first character in a pattern and a $ if it appears as the last character

41.13. expect common patterns common example

match any numbers: -1 0.35 22.2 …​

expect -re "-?(01\[1-9J\[0-9J*)?\\.?\[0-9J*"

a simpler but may not be "precise" one: P189

"^\[0-9]+$"

match a return , P112

expect -re "(.*)\n" {
}

match just one line from command output

expect -re "\n(.*)\r"       #(P119)
expect -re "(\[^\r]*)\r\n"  #(P133)

so this looks precise and enough:

expect -re "\n(\[^\r]*)\r

per my test sometime this works better:

expect -re "(\[^\n]*)\r\n"

so to match 1st/2nd/3rd line of command output:

send "cat a.txt\r"            (1)
match the 1st line of output
expect -re "\r\n(\[^\n]+)\r\n"
            ---- ------- ----
            (2)
                   (3)
                          (4)

1. user type a cmd "cat a.txt" and hit return

2. the cmd "cat a.txt" will be echoed back, and "return" is converted to the first return+newline \r\n

3. 1st line in the real output of the cmd

4. newline for the 2nd line of output

or do it line by line:

send "cat $infile\r"
#absorb cmdline itself
expect -re "^(\[^\n]*)\r\n"
#first line
expect -re "^(\[^\n]*)\r\n"
#second line
expect -re "^(\[^\n]*)\r\n"

collect user input info from stdin :

expect_user {
    -re "(\[^\n]+)\n" {
    }
}

Note	use \n here instead of \r?

match 2nd line of command output

these code works well:

send "cat $infile\r"    (1)
#absorb the echoed "cat filename.txt" line
expect -re "^(\[^\n]*)\r\n"     (2)
#check the 2nd line, see if any error happened
expect -re "^(\[^\n]*)\r\n" {   (3)
    switch -regexp -matchvar match -- $expect_out(1,string) {
        "No such file" {return 1}
    }
}

set timeout -1
expect {
    -re "(^\[^\n]*)\r\n\[^\n]*$pattern_common_prompt" {         (5)
        puts $out "$expect_out(1,string)"
        send_user "."
        close $out
    }
    -re "^(\[^\n]*)\r\n" {                                      (4)
        puts $out $expect_out(1,string)
        send_user "."
        exp_continue
    } timeout {
        puts "timeout!"
        close $out
    }
}

1. cat a whole file

2. ignore the 1st line, which will always be the echoed command line user typed in. to ignore it just match the very 1st line of the "cat file" output and do nothing

3. check the 2nd line, and see if there are errors

4. for each single new line, extract the content, ignoring both \r\n and save to file via puts. (puts will append a \n again, so this effectively just removes the \r)

retrieve the prompt

send "\r"
expect -re "\r\n(\[^\r]+)$" {
}

capture a "prompt" , works good most of the time. modified from P121

set options(pattern_common_prompt) "(% |> |# |\\\$ |%|>|#|\\\$)$"

simplied as:

set options(pattern_common_prompt) "((%|>|#|\\\$) ?)$"

further simplified (one less \):

set options(pattern_common_prompt) "((%|>|#|\\$) ?)$"

Tip	reason \\$ is same as \\\$ here, is because: TCL won’t do substitution to a single $ char - it does not look a valid var - a var with no name…​ This only applies to a single $ as a special case.

41.14. expect limitation and workaroud

it’s hard to diff between "timeout" to a pattern match, and a stalled "character flow" with the native expect command.

expect {
    -re ”$pattern" {
    }
    timeout {
    }
}

when "timeout" fires, it only indicates a no match to the pattern $pattern.

workaround

crtc "slow_mode"

41.15. expect-send pair essentials

• use expect, send "must" be used together, otherwise out-of-order screen output will occur.

"send" command is too fast - it just send the string out and immediately return, without awaiting for any output! actually in the case of interaction with remote machine with telnet/ssh, the cmd display is always too slow to be printed in the right place - right after the "send", and before other expect statements are executed.

an "expect" after the "send", is important to slow down the send ,and to "sync" the command sending and cmd output displaying!

see "test1.exp".

send_user "will send a timeout message after expect match"   ;#(1)
expect {
    -re "$pattern" {
        send -i $spawn_id "#this is an timeout\r"       ;#(5)
        puts "within expect, sleep 5s "                 ;#(2)
        sleep 5
        puts "within expect, send 2 timeout again"      ;#(3)
        send -i $spawn_id "#this is an timeout\r"       ;#(7)
        send -i $spawn_id "#this is an timeout\r"       ;#(8)
    }
}

puts "will send show version after expect match"        ;#(4)

expect {
    -re "$pattern" {
        send_user "got a match, send show version now\n"        ;#(6)
        eval {send -i [set session] "show version\r"}           ;#(13)
    }

41.16. expect vs regexp vs switch

regexp "$pattern" "$string" match substring1 substring2 ..

switch -exact -- $string {
    "$pattern" {
        $action
    }
    "$pattern" {
        $action
    }
    default {
        $action
    }

• very alike.

• same internal pattern matcher

• expect_out(0,string) = match

• expect_out(1,string) = substring1

• expect_out(2,string) = substring2

41.17. exp_continue

VERY useful, I used it a lot…​

#IMPORTANT: absorbing any possible extra prompts or whatever
set timeout_old $timeout
set timeout 1
expect -i $process -re ".+" {exp_continue -continue_timer}  #<------
set timeout $timeout_old

expect {
-i "$user_spawn_id" ;#{{{4}}}
    -re [CONST $key_interact] {   ;#{{{5}}}
        myputs2 "session:\\\[$router\\\]:you typed $key_interact key\
            here..."
        myputs2 "\nyou have the control now...\n"
        mycatch "stty -raw"
        #set oldmode [stty -raw]
        myputs "myexpect return RETURN_EXPECT_USER_INTERUPT"
        return "RETURN_EXPECT_USER_INTERUPT"
    }
    -re ".+" { ;#{{{5}}}
        puts "you typed something here...type $key_interact if you\
            want to interupt crtc..."
        exp_continue        #<------
    }

-i $process ;#{{{4}}}
    -re {$pattern_more} {   ;#{{{5}}}
        exp_send -i $process $pattern_more_key
        exp_continue        #<------
    }
}

41.17.1. absorbing extra chars

This is a very useful and frequently used feature: absorbing any possible extra prompts or whatever:

set timeout 1
expect -i $session -re ".+" {exp_continue -continue_timer}

without this sometime expect - send - expect sequence will lose synchronization.

41.18. expect -notransfer

change the default behavior: never clear data from internal buffer, even after a match

41.19. expect null

null: all zero byte: "00000000"

41.20. expect_user

• special case of expect/send -i $user_spawn_id

expect_user vs. gets stdin

-echo -reset -exact "!c" {
    ......
    set read_stdin [gets stdin]
    ......
}

vs.

p347

-echo -reset -exact "!c" {
    ......
    expect_user {
        -re "..." {
        }
        -re "..." {
        }
    }
    ......
}

41.21. expect -brace

P160: interesting discussion: how expect "guess" the "role" of its parameters:

41.21.1. default "rule" of guess

• one line arguments are treated as one pattern

• multiple line arguments, are treated as a pattern-action list

the pattern action list is usually put inside of a brace

Example 23. multiple argument, using continuation

expect \
patl actl \
pat2 act2 \
pat3 act3

Example 24. single argument, using braces

expect {
    patl actl
    pat2 act2
    pat3 act3
}

41.21.2. inconsistencies

a one line pattern, is treated as pattern-action lists, because the pattern itself "looks like" a pattern-action list (due to the existence of \n)

Example 25. single line pattern (wrongly) treated as a pattern-action list

expect "\npatl actl \npat2 act2 \n"

set pattern "\npat1 act1 \npat2 act2 \n"
expect $pattern

what are the pattern and action?

1. one single pattern containing a list of strings, no action? (no)

2. same as previous two example, still treated as a list of patterns actions, instead of a single pattern? (yes)

test:

expect [~]expect "\npat1 act1 \npat2 act2 \n"
pat1        #<------input
invalid command name "act1"         #<------error
while evaluating {expect "\npat1 act1 \npat2 act2 \n"}

even within "brace", a one line pattern-action list is still treated as one single pattern, just because it’s in one line, and so "looks like" one pattern

Example 26. one line pattern-action list (wrongly) treated as a single pattern

expect [~]set timeout 180
180
expect [~]set pattern2 {pat1 act1 pat2 act2}
pat1 act1 pat2 act2
expect [~]expect $pattern2
pat1 act1 pat2 act2         #<------input
expect [~]                  #<------match and return

41.21.3. force a single argument treated as a single pattern

In order to force a single argument to be treated as a pattern, use the -gl flag

Example 27. guarantee a single pattern

set pattern "\npat1 act1 \npat2 act2 \n"
expect -gl $pattern

41.21.4. force a single arugment treated as a list of patterns

solution is to use -brace

Example 28. -brace

expect [~]set timeout 180
180
expect [~]set pattern2 {pat1 act1 pat2 act2}
pat1 act1 pat2 act2
expect [~]expect -brace $pattern2
pat1
invalid command name "act1"
while evaluating {expect -brace $pattern2}
expect [~]

41.22. expect_before/expect_after

42. send

exp_ cmd	Functions
send	send message to current spawned process
send_user	send message to stdout
send_tty	send message to /dev/tty
send_log	send message to file (log_file)
send_error	send message to stderr

send_tty/exp_send_tty vs puts

send_tty won’t be redirected by shell…​maybe useful in some cases.

send_user/exp_send_user

• special case of send -i $user_spawn_id

• will be redirected if whole script got redirected (send_tty won’t)

• allows logging of output through log_file

• prefered over tcl "puts" (P182)

• commonly used with "log_user 0"

• send_user -raw : disable the output translation (\n→\r\n)

send_log/exp_send_log

send to files opened by:

• log_file

• exp_internal

42.1. send vs. puts

P284

• puts is used for communicating with files opened by Tel’s open command

• send works with processes started by spawn

• send: controlled indirectly by log_user

• puts: by default terminate lines with a newline, unless using -nonewline, This is convenient when it comes to writing text files

• send: Most interactive programs either read characters one at a time or look for commands to end with a \r. intead of newlines.

• send: -raw is useful in raw mode

use puts to emulate send_user:

one observation is, when iteract is slow down, because of too many patterns, all "send_*" will "appear" slow down the same. this includes send, send_user, send_tty, send_error.

puts, under this specific scenario, can be used to emulate what send_user does.

proc myputs3 {msg} {    ;#{{{2}}}
    puts -nonewline "$msg\r"
}

43. spawn

it looks, after spawn in proc, the spawn_id (or at least any varible that hold the value of it) has to be global. otherwise not working..

44. email

mail (expect book)

set to pings@juniper.net
set from test@test.net
set subject "test"
set body "test"

exec mail $to << "From: $from
To: $to
From: $from
Subject: $subject
$body"

sendmail

/usr/lib/sendmail -t < body.txt

body.txt:

To: bob@example.com
From: tim@example.com
Subject: Example of conversation threading
In-reply-to: <put Message-ID of previous mail here>

Body text here

tested this works:

set to "pings@juniper.net"
set from "no-reply-pings@juniper.net"
#this does not seem to be working
set replyto "test@test.net"
set subject "test"
set body "test"

exec sendmail -t << "To: $to
From: $from
Subject: $subject
In-reply-to: $replyto
$body"

Note	"TO: .." has to be in the very first line. "In-reply-to" does not work

45. \r\n and stty mode

45.1. all about \r\n

basis

• \r means "return", "carriage return", CR, ascii 13,0x0d, mac use as return, displayed as ^M in unix

• \n means "line feed", LF, ascii 10,0x0a, unix use as return,

• "new line" representation: to windows: \r\n, unix:\n, mac:\r

hit enter:

ping@ubuntu1:~$
ping@ubuntu1:~$

tshark:

183 Jun  5, 2016 15:35:15.279349000     10.85.47.3      10.85.4.32      \x0d
Jun  5, 2016 15:35:15.279842000 10.85.4.32      10.85.47.3      \x0d\x0a,ping@ubuntu1:~$

type "pwd" hit return:

ping@ubuntu1:~$ pwd
/home/ping
ping@ubuntu1:~$

tshark:

464 Jun  5, 2016 15:42:18.007030000     10.85.47.3      10.85.4.32      p
Jun  5, 2016 15:42:18.007436000 10.85.4.32      10.85.47.3      p
Jun  5, 2016 15:42:18.104298000 10.85.47.3      10.85.4.32      w
Jun  5, 2016 15:42:18.107340000 10.85.4.32      10.85.47.3      w
Jun  5, 2016 15:42:18.248993000 10.85.47.3      10.85.4.32      d
Jun  5, 2016 15:42:18.249344000 10.85.4.32      10.85.47.3      d
470 Jun  5, 2016 15:42:18.478308000     10.85.47.3      10.85.4.32      \x0d
Jun  5, 2016 15:42:18.478767000 10.85.4.32      10.85.47.3      \x0d\x0a,/home/ping\x0d\x0a,ping@ubuntu1:~$

router: hit enter:

labroot@seahawks-re0>

labroot@seahawks-re0>

tshark:

279 Jun  5, 2016 15:39:20.148506000     10.85.47.3      172.19.161.201  \x0d
Jun  5, 2016 15:39:20.149195000 172.19.161.201  10.85.47.3      \x0d\x0a,\x0d\x0a
Jun  5, 2016 15:39:20.149595000 172.19.161.201  10.85.47.3      labroot@seahawks-re0>

in summary

• in line mode, term driver do some work "behind the scene", making things more convenient:

  • input: you hit return \r, translated to \n, then echoed.

  • output: \n to \r\n (so your input \r eventually triggered a \r\n)

  • to make it , use expect_user "\n", or expect_user "\r\n"

• in raw mode, term driver skipped some work: no translation/special char process (still echo?):

  • input: your \r remains \r, to match it use expect_user "\r"

  • output: no \n to \r\n translation

  • but, to make it easier, send will still do \n → \r\n translation

  • use -raw to disable this send behavior

some typical usage examples:

expect "1st line pattern\r\n2nd line"
send_user "keyword pattern found!\n"
send "pwd\r"

45.2. raw/cooked mode

mode

general             most common form
====================================
line-oriented       cooked
cha-oriented        raw

line mode

• term driver do a lot of work to make things more convenient:

  • buff all keystrokes until a return is pressed P197

  • echo

  • some translations

• when buffered, special chars are interpreted to do special things: backspace to delete, etc

• this provide a "minimally intelligent user-interface" and drastically simplified most programs

• input(keystroke):\r → \n translation P198

• output(display): terminal driver translate: \n to \r\n (P198) in line mode

line mode examples P197

send "Enter your name: "
expect_user "\n"

user can fix typo and hit return when done

raw mode

• expect_user does not wait for a return, and will try match immediately after keystroke

• \r works fine

• input: no \r → \n translation

  send "Enter your name: "
  expect_user "\r"

• ouput: \n just move to newline without "return"

• output: "new line becomes line feed" P345 (meaning no \n → \r\n output translation)

• output: send_user automatically translate newline to \r\n P345(P197), -raw disable this

other nodes: in expect/send

• in send, use \r to represent "hit a return"

• in expect(including expect_user) cmd, terminal driver do translations:

  • input(keystroke) translate: return (\r) to \n (new line) only in line mode (P192, P198)

• in send(send_user)

  • output(display) translate: translate \n to \r\n also under raw mode

  • output translation can be disabled by -raw

45.3. mode changing: stty

• expect stty calls native stty command in the system, so any native stty parameters can be provided

• expect stty provides some extra parameters for user’s convenience

  stty echo|-echo
  stty raw|-raw|cooked|-cooked

Tip	and when using these parameters, expect stty won’t call system stty.

• possibility of losing chars while switching modes

• should be executed during time when user is NOT typing (P199)

  stty raw ;# Right time to invoke stty
  send "Continue? Enter y or n: "
  stty raw ;# Wrong time to invoke stty

inconvenience of no "stty -info"

this is the best practice so far: always flip it back first before flip it, this will ensure tracking of oldmode won’t be messed up.

to backup oldmode

if [info exists oldmode] {eval stty $oldmode}
set oldmode [stty -echo]

to recover oldmode

eval stty $oldmode

otherwise considering this:

#backup oldmode
set oldmode [stty -echo]
#backup oldmode again
set oldmode [stty -echo]
#recover old mode
eval stty $oldmode

initially say stty parameter is "raw, echo", first oldmode is set to "raw echo", the second oldmode will be set to "raw -echo".

example of "mode dependent commands" P345

system cat file
exec kill -STOP [pidl
expect -re "(. *) \n"

46. match_max and full_buffer

match_max

defines the size of the buffer (in bytes) used internally by expect.

While reading output, more than 2000 bytes can force earlier bytes to be "forgotten". This may be changed with the function match_max. (Note that excessively large values can slow down the pattern matcher.) If patlist is full_buffer, the corresponding body is executed if match_max bytes have been received and no other patterns have matched. Whether or not the full_buffer keyword is used, the forgotten characters are written to expect_out(buffer).

Note

to understand this knob, you have to understand how expect cmd works internally: spawned process generate output in the form of "chunk" by "chunk", whenever expect received one chunk of output, it will put in an internal buffer and scan the whole buffer to match patterns. if no patterns found and more chunks of strings come in, expect will re-scan the whole accumulated buffer to match patterns. if this whole internal buffer grows bigger and bigger (because of no match) and exceeding the configured "match_max" size, expect will "throw away" the older strings and only keep the latest "match_max" size of buffer for further pattern match - this esentially limits the performance impact introduced by a huge buffer (because of no match) to a certain, controllable extent.

P166 section "Pattern Debugging" , provided a detail illustration with exp_internal.

some related performance fine-tune method: P151

• match incoming strings "line by line", and append each new line into a buff

• design a match to skip some predictable but unwanted strings blocks, and then use a second match to search in a much smaller rest part of the input

full_buffer

when full, all internal buffer will be moved to expect_out

buffer slow users input, and send to process in a bulk, every 3s or full_buffer reach, whichever comes first. great! P152

set timeout 3
while 1 {
    expect_user
    eof exit
    timeout {
        expect_user "*,,
        send $expect_out(buffer)
        full_buffer {send $expect_out(buffer)}
    }
}

47. log_user/log_file

log_user

log_user -info|0|1

suppress all output from spawned process, plus the echo from "spawn" command

• By default, the send/expect dialogue is logged to stdout (and a logfile if open).

• The logging to stdout is disabled by the command "log_user 0" and reenabled by "log_user 1".

• Logging to the logfile is unchanged

• The -info flag causes log_user to return a description of the most recent non-info arguments given.

• commonly used with "send_user" (after log_user 0)

Note

it seems, log_user rely on "expect" to work, without "expect", the output won’t be suppressed…​ log_user 0 #<------(1) send_verbose "copying\n" send "cat > $outfile\r" set fp [open $infile r] while {1} { if {-1 == [gets $fp buf]} { break } else { } send_verbose "." send -- "$buf\r" } if {$verbose_flag} { send_user "\n" ;# after last "." } send_user "now send eof\n" send "\004" ;# eof send_user "now close fp\n" close $fp expect -re "$prompt" #<------(2) log_user 1 suppress spawned process output to screen(stdout) test shows: without this the log_user above doesn’t work…​

log_file

• "ANY" output from spawned process

  • including all control chars

  • not record "send", but will record if echoed from spawned process

  • so password won’t be recorded since it is not echoed

• any diagnostics info expect itself generated

• "raw" log info - exactly what user is doing

• but not including Tcl’s cmd output - puts

  • info from send_user will be recorded - one main diff with puts!

• by default, it looks the log file generated this way will contains '\r\n'

  • \n will be displayed as a new line

  • \r will be displayed as a ^M in vim

  • if the desire is not to record \r, then use explicity file puts, instead simply log_file. see example and explanation in P257.

flags:

• -noappend not to append, but to overwrite

• -a ignore "log_user 0"

• -info return current status

  #!/usr/bin/expect
  log_user 0                  ;#<------disable default disaplay from spawned process
  spawn $env(SHELL)
  stty raw -echo              ;#<------put terminal in raw mode
                              ;#so no need "press" \r"
                              ;#no output that user can see

  set timeout -1
  set fp [open typescript w]  ;#<------open a file for log

  expect {
      -re ".+" {
          send -i $user_spawn_id $expect_out(buffer)
          exp_continue
      }

  eof exit

  -i $user_spawn_id -re "(.*)\r" {            ;#<------strip `\r`
      send -i $spawn_id $expect_out(buffer)   ;#<------send (w/o \r)
      puts $fp $expect_out(1,string)          ;#<------log w/o \r
      puts $fp [exec date]
      flush $fp
      puts "get -$expect_out(1,string)- in slashr"
      exp_continue
  }

      -re ".+" {
          send -i $spawn_id $expect_out(buffer)
          puts -nonewline $fp $expect_out(buffer)
          flush $fp
          puts "get -$expect_out(buffer)- in dotplus"
          exp_continue
      }
  }

47.1. pending issue

there is no known/existing/good way to do log cleanup: interpret all contrl chars. this works close, but seems still buggy

cat seahawks-re0.log | col -bp | less -R
cat seahawks-re0.log | iconv -t utf-8 | col -bp | less -R

48. signal

Name        Description
===================================
SIGINT      interrupt               ^c
SIGTERM     software termination
SIGQUIT     quit                    ^\
SIGHUP      hangup
SIGKILL     kill
SIGPIPE     pipe write failure
SIGSTOP     stop (really "suspend")
SIGTSTP     keyboard stop
SIGCONT     continue
SIGCHLD     child termination
SIGWINCH    window size change
SIGUSRl     user-defined
SIGUSR2     user-defined

SIG_IGN SIG_DFL

spawn -ignore

SIGINT

• triggered by ctrl-c under cook mode, changable via stty

• under raw mode, ctrl-c will not trigger SIGINT, but instead be sent literally to spawned process

SIGTSTP/SIGSTOP/SIGCONT

ctrl-z ⇒ SIGTSTP (not SIGSTOP, which can’t be caught) fg ⇒ SIGCONT

man 7 signal:

SIGCONT   19,18,25    Cont    Continue if stopped
SIGSTOP   17,19,23    Stop    Stop process
SIGTSTP   18,20,24    Stop    Stop typed at terminal

The signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored.

need confirm:

"Control-Z sends SIGTSTP, not SIGSTOP. An important difference between them is that programs can catch or ignore SIGTSTP, but not SIGSTOP. Programs may catch TSTP and perform cleanup operations before suspending execution, but STOP causes the process to stop without any notice. –" http://superuser.com/questions/287428/whats-the-difference-between-s-and-z-inside-a-terminal

trap all signals

for {set i 1} {$i<=[trap -max]} {incr i}
    catch {trap $handler $i}
}

49. interact

exit -onexit

• continues after a match (not like expect)

• continue to shuttle chars back and forth between user and process

49.1. interact_out

• no interact_out(buffer)

• interact_out(0,string) matched strings

• interact_out(1,string) value captured in 1st parenthesis

• interact_out(0,start)

• interact_out(0,end)

49.2. interact in a loop

P339

49.3. nested interact (interact under interact)

P338

49.4. interact default action: interpreter

P341

these are same, but only if X is the last pattern

interact X
interact X interpreter

testing shows,

• press + will enter interpreter, then

• if type "return" will return to remote device

• else if type "exit" will exit script

question:

how does this work: P9

while 1 {
    expect {
        eof {break}
        "UNREF FILE*CLEAR\\?" {send "y\r"}
        "BAD INODE*FIX\\?" {send "n\r"}
        "\\? " {interact + }
    }
}

49.5. interact on eof

default/implicit action is "return"

49.6. interact timeout

• diff value can be used for input from user or spawned process: P343

  interact {
      timeout 10 {
          send_user "Keep typing-we pay you by the character!"
      }
      -o
      timeout 600 {
          send_user "It's been ten minutes with no response.\
              I recommend you go to lunch!"
      }
  }

• anti_idle_string candicates: P343-P344

  • " "

  • " \177"

  • ctrl-g

  • null

maybe the best will be to track if ^vim is ever entered, then choose diff one as anti_idle_string!

this doesn’t work:

interact {
    "!Q" exit
    timeout $timeout {
        puts "timeout $timeout!"
        incr timeout
    }
}

will prints below, but every 2s exactly:

timeout 2! timeout 3! timeout 4! timeout 5! timeout 6!

49.7. implicit spawn_id and -o

             expect/interact
                |
                |   <------
                |     -o "abc"
                |
                |
                | (-i $spawn_id)
 o             telnet
                |
/|\  -----------+----------//--          remote machine
/ \             |
                |
                |
      ====================>

interact {
    "eunuchs" {send "unix"} \      user->      process
    "vmess" {send "vms"}     X       --------->
    "dog" {send "dos"}      /
    -0
    "unix" {send_user "eunuchs"} \ user      <-process
    "vms" {send_user "vmess"}     X  <---------
    "dos" {send_user "dog"}      /
}

49.8. -i (explicit spawn_id)

-i $spawn_id

             expect/interact
                |
                |
       ----->   |
       "!d"     |
                | (-i $spawn_id)
 o             telnet
                |
/|\  -----------+----------//--     spawned process
/ \             |
                |
                |
      ====================>

interact {
    $user_key $act
    -i $proc1 {
        -re $pat1 act1
        -re $pat2 act2
    }
}

49.9. -u

                expect/interact
                   |
                   |
          ----->   |
                   |
                   | (-i $spawn_id)
                   |  telnet
                   |
spawned -----------+----------//--     spawnd process
process            |
                   |
                   |
         ====================>

49.10. -input -output

• default

  interact -input $i -output $01 -output $02
  interact -input $i -output "$01 $02"

• default

  • If the first -input is omitted, user_spawn_id is used.

  • If the first -output is omitted, spawn_id is used.

  • If the -output after the second -input is omitted, user_spawn_id is used.

• with -u

  interact -u $proc -output $out -input $in

this is same as:

interact -input $proc -output $out -input $in -output $proc

• multiple -output or -input (kibitz)

  interact
      -input $user_spawn_id   -output $process
      -input $userin          -output $process
      -input $process         -output $user_spawn_id
                              -output $userout

• combine multiple input/output

there 2 are the same:

interact -input $i -output $o1 -output $o2
interact -input $i -output "$o1 $o2"

The following command takes the input from i1 and i2 and sends it to 01.

interact -input "$i1 $i2" -output $o1

• two -input flags in a row with no -output in between causes the first input source to be discarded. This can be quite useful for the same reasons that it is occasionally handy to redirect output to /dey/null in the shell.

Using these shorthands, it is possible to write the interact in kibitz more succinctly:

interact {
    -input "$user_spawn_id $userin" -output $process
    -input $process -output "$user_spawn_id $userout"
}

49.10.1. kibitz

basic usage: p355

kibitz user1
kibitz user1 tty1
kibitz user1 vim
kibitz user1 kibitz user2

kibitz -noproc -tty ttyab user1

Note	this doesn’t work well:

kibitz user1 -noproc -tty ttyab

how it works

interact
    -input $user_spawn_id   -output $process
    -input $userin          -output $process
    -input $process         -output $user_spawn_id
                            -output $userout

49.11. -iwrite

-iwrite flag controls whether the spawn id is recorded, in this case, to interact_out (spawn_id) .

interact {
    -input  "$user_spawn_id $userin"
    -iwrite "foo" {actionl}
            "bar" {action2}
    -iwrite "baz" {action3
}

The -iwrite flag forces the spawn_id element of interact_out to be written P361

49.12. indirect spawn_id

2 method of dynamic expect/interact to switch between multiple spawn_id:

• use a expect/interact loop, use normal $spawn_id value, change -i varible value, then continue to next expect/interact execution

• in current expect/interact, use indirect spawn_id, any change will take place right away.

indirect spawn_id make it possible to dynamically update spawn_id inside of expect/interact, very useful!

examples in book:

interact {
    -input inputs -iwrite eof {
        set index [lsearch $inputs $interact_out(spawn_id)]
        set inputs [lreplace $inputs $index $index]
        set outputs [lreplace $outputs $index $index]
        if {[llength $inputs]==0} return
    } -output $process
    -input $process -output outputs
}

• When inputs and outputs are modified, interact modifies its behavior to reflect the new values on the lists.

• The length of the input list is checked to see if any spawn ids remain.

• If this check is not made, the interact will continue the connection.

• However, with only the process (in process) participating, all that will happen is that the output of it will be discarded.

• This could conceivably be useful if, for example, there was a mechanism by which spawn ids could be added to the list (perhaps through another pattern).

examples in crtc:

set myinteractcmd [subst -nocommands {\n\
    interact {\n\

-input $user_spawn_id\n\
    $myinteract_user_input_patterns\n
-output process_output\n\

-input kibitz_spawn_id
    ......
-output process_output\n\

        -input process_input\n\
            $myinteract_process_input_user_patterns\n\
            $myinteract_process_input_patterns_misc\n\
            $myinteract_process_input_patterns_static\n\
        -output $user_spawn_id\n\
            $myinteract_user_output_patterns\n\
        -output kibitz_spawn_id\n\
    }\n\
}]

eval $myinteractcmd

the "kibitz_spawn_id" and "process_input" "process_output" here all indirect. so from an user input "!k", trigger any change of these vars, will change the interact behaviors, dynamically.

49.13. -reset

49.14. puts under interact

this won’t give good format:

puts "interact: detected event\n-$event-!!"

this won’t either:

puts "interact: detected event:"
puts "-$event-"

tried send_tty, send_user, not change…​

-reset seems resolved it.

49.15. inter_return vs return

inter_return causes interact to cause a return in its caller. For example, if "proc foo" called interact which then executed the action inter_return, proc foo would return.

50. expect vs. interact

these are all different technique to deal with certain situations:

50.1. call expect_user under interact

to collect user info

50.2. call expect under interact

an interesting effect: absort some text output that will otherwise display in the screen…​

P?

crtc:

myinteract : !K -> interact_K -> inviteremote ..

proc inviteremote {} {        ;#{{{4}}}
    expect {
        "is not logged in" {
            send_user "$kibitz_user1 is not logged in yet, check and\
                try again later!"
            set inkibitz 0
        }
        -re "Escape sequence is \\^\\\]" {

            send_user "kibitz succeeded!\n"
        }
    }
}

result:

who are you going to invite?
user1
will invite user1 ...
kibitz succeeded!

the normal interaction should be like:

ping@ubuntu47-3:~$ kibitz -noproc user1
asking user1 to type:  kibitz -23054
write: user1 is logged in more than once; writing to pts/69
Escape sequence is ^]

50.3. call interact under expect

to allow user takes control for a while, they typically type some strokes to return back to expect

expect {
    ...
    script/expect control
    ...
    interact \\ {
        ...
        user control
        ...
    }
    ...
    script/expect control
    ...
}

50.4. call interact under interact (nested interact)

51. -i option

• expect/send/interact

• wait/close/match_max

• parity/remove_nulls

  spawn_id
  any_spawn_id
  error_spawn_id
  user_spawn_id
  tty_spawn_id

52. Expect specific cmds

52.1. interpreter

When Expect is interactively prompting for commands, it is actually running a command called interpreter.

When Expect is interactively prompting for commands, it is actually running a command called interpreter.

When Expect is interactively prompting for commands, it is actually running a command called interpreter.

52.2. prompt1/2

53. Expect errors

a typo in code:

expect {
    -i $spawn_id {{{4}}}    #<------
    -re {$} {
        puts "found a match to -$pattern1-"
        puts "expect_out(buffer) looks -$expect_out(buffer)-"
    }
    timeout {
        puts "timeout when looking for pattern"
    }
}
exp_internal 0

this expect will timeout with below debug msg:

(spawn_id exp6) match glob pattern "{{4}}"? no
"$"? no
expect: timed out

running under unix at

send_tty: cannot send to controlling terminal in an environment when there is no controlling terminal to send to!

54. "screen" sharing

• shell redirection

  crtc -a "set double_echo 1" myrouter > temp.log
  tail -f temp.log

• log_file

  tail -f router.log

• internal redirection

  catch {exec echo "[set interact_out(0,string)]" >  [set dest_tty]}

• kibitz

55. debug Expect script

55.1. debug 1

55.2. trace

55.3. exp_internal

exp_internal 0          no diagnostics
exp_internal 1          send pattern diagnostics to standard error
exp_internal -f file 0  copy standard output and pattern diagnostics to file
exp_internal -f file 1  ..and send pattern diagnostics to standard error

55.4. expect debugger

tcl extentions

56. sqlite: most popular tcl extension?

sudo apt-get install sqlite3 libsqlite3-tcl

---

1. see config file for other alternative way to do the same thing.

2. of course, better run chmod 755 printver.sh to make the file executable

3. or 1TBS style according to wikipedia

4. actually the config file is just a TCL source code,

5. if -x being used, the issue will be defined as all conditions must be met

6. it’s not hard to eliminate the gap and sync the local time display with the remote, but I’ll add this ability later

7. and it supports some small features I don’t see from other tools:)

8. even this tutor doc

9. yes,the characters will be delivered to the remote device only when the input does not match with any of the pre-defined keybinding