PAN OS - Backing up through XML API

[ghost]

PAN OS comes with an excellent API that allow an extraction of the configuration in XML format. The XML format is the only good way to backup a PAN OS device. The panos.rb is good to know when a device was modified, but can't be used to restore a device from scratch.

It will be great to add a generic device type that allow a backup from an URL and not standard ssh/telnet. The only needed variable in the case of PAN OS is the URL to download. The security is provided by key gen that needs to be included in the URL.

[ytti]

It shouldn't be very hard to add http as input method, so that we could fetch configs with it. I wonder if it truly can be generic. Like would HTTP API always be single URL that returns whole config, or could it perhaps require parsing the output of single GET to figure out what to get next? It may be that generic model won't work.

Essentially to get this working, we'd add http.rb here https://github.com/ytti/oxidized/tree/master/lib/oxidized/input then specify in model cfg :http block for it. Model could also be http.rb if we're happy with generic model for http. But if my fear above is justified, we need to still do per model ones for http.

If we need to do per model ones, and same model must support ssh and http then we need slight infrastructure change, because we need to discriminate cmd blocks, to determine which is for httpand which is for ssh, but that should't be too hard. It could be something like:

cmd 'url', :http do |cmd|
   # cmd is output of get, only run for :http
end

cmd 'command', [:ssh, :telnet] do |cmd|
   # cmd is output of command in shell, only run for :ssh and :telnet

This isn't novel request, we've added ftp support earlier here bf93722 if this sort of solution is fine, where given model only needs http, this is easy. If given model needs http AND ssh/telnet, it's more dicy.

Also as I don't have any PanOS devices, maybe @rixxxx wants to chime in too.

[ytti]

What is the status here?

[sau70networkadmin]

bump -- would also like this feature. As FTBZ mentioned, the PanOS method is a single URL containing an API key which returns the whole config as xml. Details can be found here: https://live.paloaltonetworks.com/t5/Management-Articles/How-To-Backup-of-Config-Files-Periodically-wiithout-Panorama/ta-p/77312 Thanks for oxidized! :)

[athompson-merlin]

(There's somewhat good news buried halfway through this. Keep reading!)

This probably isn't the best place to document this, but I can't find a really good spot, so, here we go.

On my PA7050 (PANOS v9.1.6) XML output can theoretically also be accomplished by first sending "set cli config-output-format xml", at which point "show config running" is supposed to give you a dump of the XML config file.

Unfortunately, this appears to simply not work for "show config running" which continues to emit JSON-format config.
What DOES work is to execute "show" from inside "configure" mode:

athompson@Nyx> 
athompson@Nyx> set cli config-output-format xml
athompson@Nyx> configure
Entering configuration mode
[edit]
athompson@Nyx# show
<response status="success" code="19">
  <result total-count="1" count="1">
    <deviceconfig>
      <system>
       [...lots and lots more XML...]
      </password-complexity>
    </mgt-config>
  </result>
</response>
[edit]
athompson@Nyx# exit
Exiting configuration mode
athompson@Nyx> 

I'd be happy to report the fact that show config running doesn't honour set config-output-mode as a bug to PA if I thought they'd care for even a nanosecond, but they've demonstrated that they don't care about showstopper bugs, never mind cosmetic issues (support has outright refused to file bug reports for us in the recent past).

In any case, this approach is also suboptimal because the userid used must have global configuration rights in order to enter "configure" mode in the root (non-vsys) context, and that's really bad practice.

The good news is that there is a command that DOES affect the output of show config running, and that is set cli op-command-xml-output on

With that enabled:

athompson@Nyx> 
athompson@Nyx> set cli op-command-xml-output on
athompson@Nyx> show config running 
<response status="success"><result><config version="9.1.0" urldb="paloaltonetworks">
  <devices>
    <entry name="localhost.localdomain">
      <vsys>
       [...tons more XML...]
    </password-complexity>
  </mgt-config>
</config></result></response>
athompson@Nyx> 

I do not know what version of PANOS introduced op-command-xml-output so I don't know how useful this is generally. Looks like at least 9.0, but I cannot yet confirm that.

Lastly, while my output looks sane, there's this in the 10.0 release notes:

PAN-149008 | Fixed an issue where the CLI command Show config running following the CLI command set cli op-command-xml-output on produces an unreadable output.

so... ????

I think it's probably worth putting into PANOS-specific notes, at the very least, and possibly adding a config flag that toggles it on if desired. It's much more difficult for humans to read than the JSON or SET-style output, so some people may prefer the default json formatting.

[pv2b]

Cool find! That's definitely some impressively creative abuse of the functionality PanOS offers. :-)

Ultimately though, my (biased) opinion is that just backing up through the HTTPS API is probably a better, more sustainable method long term, as per PR #2360 that I put in just a few days ago. (That said, if you had posted this a week ago I probably wouldn't have bothered with my other solution.)

[athompson-merlin]

Oh, you're right about that - the API is a cleaner way to do it, for sure, esp. long term, given that CLIs make notoriously unstable APIs. There's a reason Ansible, e.g., is pushing the API route hard.

However, there are various people (including me, in some spots) who can't do the HTTP APIs for one reason or another, so I think it's still worth documenting. There's documentation on how a user can extend the standard functionality, for example, on IOS, I think something similar would be appropriate. But... perl, no problem, python, maybe, ruby? forget it. I can do the text, but not code.

[mortzu]

There is a PR #2360

[macarpen]

@athompson-merlin (or anyone else finding this) below is a CLI based approach for pulling an XML config from PAN-OS using the workaround you found.

require 'nokogiri'

class PanOSXML < Oxidized::Model

  prompt /^[\w.@:()-]+>\s?$/

  cmd 'show config running' do |cfg|
    "<?xml version=\"1.0\"?>\n" +
    Nokogiri::XML(
        cfg.split("\n")[2..-2].join("\n")
    ).at('/response/result/config').to_xml(indent: 2)
  end

  cfg :ssh do
    post_login 'set cli pager off'
    post_login 'set cli op-command-xml-output on'
    pre_logout 'quit'
  end
end

[athompson-merlin]

I would love to try this!
In theory, I should create /etc/oxidized/model, put that code into /etc/oxidized/model/PanOSXML.rb, and then add a new host to my inventory of type PanOSXML?

[athompson-merlin]

No go, I'm consistently getting Apr 05 17:23:57 oxidized.merlinoffice.local oxidized[1745319]: node {:name=>"XXXX", :model=>"panosxml", :username=>"XXXX", :password=>"XXXX", :vars=>{:enable=>nil, :ssh_kex=>nil, :ssh_host_key=>nil, :ssh_hmac=>nil, :ssh_encryption=>nil}} raised Oxidized::ModelNotFound with message 'panosxml not found for node 99.99.99.99'

for a router.db entry of:
XXX;panosxml;XXX;XXX

I've tried putting panosxml.rb in both /etc/oxidized/modules/ and in /usr/local/share/gems/gems/oxidized-0.28.0/lib/oxidized/model/ (just to see if it would work at all).

Clearly I've missed something trivial but vital in enabling new models...
(apologies for the censoring of info, don't have a choice)

[athompson-merlin]

Ah, the problem is my Ruby is too old for nokogiri. And now begins the quest to fully reinstall Oxidized using a slightly newer Ruby, yay.

[macarpen]

In theory, I should create /etc/oxidized/model, put that code into /etc/oxidized/model/PanOSXML.rb, and then add a new host to my inventory of type PanOSXML?

It needs to go in ~/.config/oxidized/model/panosxml.rb, then use panosxml as the model name.

[athompson-merlin]

I've found that panosxml.rb works on some of my devices but not others.
As far as I can tell, it fails on Panorama (v10.x) every time, don't know why yet.
But it also fails on a PA3220 running v9.3.1, with this error in the debug output:

Apr 26 19:29:58 oxidized.merlinoffice.local oxidized[477959]: lib/oxidized/worker.rb: 1 jobs running in parallel
Apr 26 19:29:58 oxidized.merlinoffice.local oxidized[477959]: 10.0.2.215 raised NoMethodError with msg "undefined method `to_xml' for nil:NilClass
                                                              Did you mean?  to_yaml", nemesis saved
Apr 26 19:29:58 oxidized.merlinoffice.local oxidized[477959]: lib/oxidized/node.rb: Oxidized::SSH failed for nemesis

which... I'm guessing means there was a prior, uncaught XML parsing error???

The XML output does NOT work correctly from the CLI in v9.1.3 - that's the PA bug I discussed above, AFAICT - so I tried this slightly modified model that does it from configuration context:

require 'nokogiri'

class PanOSXMLcfg < Oxidized::Model

  prompt /^[\w.@:()-]+[#>]\s?$/

  cmd 'show' do |cfg|
    "<?xml version=\"1.0\"?>\n" +
    Nokogiri::XML(
        cfg.split("\n")[2..-2].join("\n")
    ).at('/response/result/config').to_xml(indent: 2)
  end

  cfg :ssh do
    post_login 'set cli pager off'
    post_login 'set cli op-command-xml-output on'
    post_login 'set cli config-output-format xml'
    post_login 'configure'
    pre_logout 'exit'
    pre_logout 'quit'
  end
end

...but with the exact same result. I've confirmed that the XML generated is well-formed, but I don't know enough about ruby to further diagnose why "to_xml" works on other targets but fails with a bizarre (to me) error on this one. Any suggestions?

[athompson-merlin]

FWIW, configure is available to read-only user, you just can't use the set command in configure context. This was surprising to me, but a happy surprise.

[pv2b]

I don't know enough about ruby to further diagnose why "to_xml" works on other targets but fails with a bizarre (to me) error on this one. Any suggestions?

The error occurs when calling to_xml. The only place that's done is here:

.at('/response/result/config').to_xml(indent: 2)

It's complaining because you can't call to_xml on nil. That suggests that .at('/response/result/config') is returning nil.

What that's trying to do is to try to dig out the configuration which is merely part of the larger XML API response. For example:

<response ...>
  <result ...>
    <config ....> <- the config is in this element!
    </config ....>
  </result ...>
</response ...>

So, whatever's happening, it's likely that the layout of the XML response is somehow different and no longer like the example skeleton below.

To get further, you'd have to take a closer look at the structure of the XML being captured.

[athompson-merlin]

Ah, found the problem. The Panorama deployment of PAN-OS v10 lacks the set cli op-command-xml-output on command altogether. My version that dumps config from configure mode also doesn't work because the XML output is structured as

<response>
  <result>
    <device-group>
       ...
    </device-group>
  </result>
</response>
<response>
  <result>
    <deviceconfig>
    etc...

it's a completely different XML structure than what PAN-OS on a firewall produces. I'll see if I can hack together a working model that accepts Panorama output.

[ciroiriarte]

Will this be included in oxidized?, Would this method be preferred over the included logic for PANOS?