PoC for piping the raw CLI output

[mirceaulinic]

Opening this PR for discussion around piping raw CLI outputs #616

Implemented only match and count so far. In case the model proposed looks good, we can move on and implement the others:

Possible completions:
  count                Count occurrences
  display              Show additional kinds of information
  except               Show only text that does not match a pattern
  find                 Search for first occurrence of pattern
  hold                 Hold text without exiting the --More-- prompt
  last                 Display end of output only
  match                Show only text that matches a pattern
  no-more              Don't paginate output
  refresh              Refresh a continuous display of the command
  request              Make system-level requests
  resolve              Resolve IP addresses
  save                 Save output text to file
  trim                 Trim specified number of columns from start of line

Of course, except things such as no-more, as anyway it is provided the complete output, or refresh.

[coveralls]

Coverage decreased (-0.4%) to 96.71% when pulling 0d42a3a on mirceaulinic:CLI-PIPE into b52e6dd on Juniper:master.

[jnpr-community-netdev]

Autobot: Would an admin like to run functional tests?

[mirceaulinic]

@jnpr-community-netdev are you sure this library is still maintained? :P

Seriously now: @vnitinv @stacywsmith any thoughts on this one-month-old PR?

[vnitinv]

@mirceaulinic Sorry for late response. Was on PTO for whole Nov. We are evaluating option to have support for pipe '|' over netconf from backend itself. Supporting cli kind of output by PyEZ will be a big burden, as with different device types output keeps changing. I hope you do understand.

[mirceaulinic]

Hi @vnitinv - thanks for looking into this!

I hope you do understand.

Sure, I understand, no problem; happy to close this PR if anything better in place.

We are evaluating option to have support for pipe '|' over netconf from backend itself.

That's very good news! It makes much more sense to be like that, however I have two questions:

1. Is this going to be supported on older versions (I would assume on Junos 12 or 13 we can't expect any)?
2. How about the other options, e.g.: count, match, except etc.? Will they be supported as well?

Cheers!

[mirceaulinic]

@vnitinv is this still on your radar? My hunch is that the solution you have exposed will never be available in older releases, most probably it will require Junos >= 16 or even >=17, which is not a comfortable constraint given that we are able to provide a workaround, without breaking the existing behaviour.

This feature is based on the feedback of many engineers working in production environments. I know the theory, but, as always, in the real-world there are more specific needs. And filtering the output is one of them.

@stacywsmith may I have also your input on this?

Thanks,
Mircea

[vnitinv]

Can one of the admins verify this patch?

[attila123]

@mirceaulinic this was more than a year ago. Are you still willing to finish this patch?
Seems like Travis reported some error.

[mirceaulinic]

Hi @attila123 - I think I'm waiting here for feedback from the maintainers... is Juniper still against doing this thing?

[lstoor]

Hi,
Is this thread still active? I have also run into "|" related problems.
Br,
Lakhavir

[attila123]

Looks like only "| display xml rpc" piping is supported, see 'cli' method documentation at

py-junos-eznc/lib/jnpr/junos/device.py

Lines 661 to 680 in 5b6c3a1

	.. note::
	You can also use this method to obtain the XML RPC command for a
	given CLI command by using the pipe filter ``| display xml rpc``.
	When you do this, the return value is the XML RPC command. For
	example if you provide as the command
	``show version | display xml rpc``, you will get back the XML
	Element ``<get-software-information>``.
	.. warning::
	This function is provided for **DEBUG** purposes only!
	**DO NOT** use this method for general automation purposes as
	that puts you in the realm of "screen-scraping the CLI".
	The purpose of the PyEZ framework is to migrate away from that
	tooling pattern. Interaction with the device should be done via
	the RPC function.
	.. warning::
	You cannot use "pipe" filters with **command** such as ``| match``
	or ``| count``, etc. The only value use of the "pipe" is for the
	``| display xml rpc`` as noted above.

.

[ntwrkguru]

Hi @attila123 - I think I'm waiting here for feedback from the maintainers... is Juniper still against doing this thing?

A year late, but I am personally opposed to this as programmatically invoking CLI is anti-pattern when there are other mechanisms to retrieve and parse the data within the NETCONF system itself. What I mean is that the CLI simply passes a NETCONF command, and any piping is doing some processing prior to returning the data to the user. display xml rpc is the proper exception, IMO, as it allows for retrieving the RPC equivalent of a given CLI command. Several of the arguments to the pipe don't make sense for programmatic access; e.g. refresh, resolve, hold, etc.

All in all, passing CLI over NETCONF is a bit of a hack, even if it is used a lot. However, this is just my 2c and I don't represent all of Juniper or even this project. :-)

[attila123]

@ntwrkguru In our use case this cli can be executed remotely by operations people in troubleshooting situations. This is exactly what the cli is meant to be used for: debugging, troubleshooting. But currently cli is there, but in a crippled version, which limits the possibilities by not allowing most piping options. Looks like now I need to implement my own cli client with direct ssh (e.g. with paramiko library).
Update: I see now in the code that it actually does not execute the cli command as cli command, but first coverts it to rpc, which can explain why devs do not want to implement all possible pipe operations.

[ntwrkguru]

@ntwrkguru In our use case this cli can be executed remotely by operations people in troubleshooting situations. This is exactly what the cli is meant to be used for: debugging, troubleshooting. But currently cli is there, but in a crippled version, which limits the possibilities by not allowing most piping options. Looks like now I need to implement my own cli client with direct ssh (e.g. with paramiko library).
Update: I see now in the code that it actually does not execute the cli command as cli command, but first coverts it to rpc, which can explain why devs do not want to implement all possible pipe operations.

And one layer down, mgd actually converts CLI to RPC on the box as well. :-) To your point, this is the pain of cultural transformation from a box-driven paradigm to an automated one.

[rahkumar651991]

Based on the above discussions, the rpc based approach doesn't sync well with the cli pipe based implementation.
Closing this pull request.