Retrieval of configs via http

[ja-frog]

Would http as a method of config retrieval be a valid addition to Oxidized?

This came up recently for us with a Cambium PMP 450i AP. These are master radios for fixed wireless service. There are only two ways to pull their configs: via http or by first signing in via telnet to generate the config file and then signing in by ftp to download it. http seems the simpler option.

[ytti]

Absolutely. You only need to create http.rb in the input directory and then have cfg :http {}in model to use it.

Unsure how to model, but one way would be same as tftp/ftp where cmd argument would be the URL you GET

I'm kinda surprised this hasn't come up.

[wk]

This has come up in a slightly different shape and was attempted in a limited fashion in #1110 and previously in #1103 (originally from #440). An HTTP input would allow elegantly closing those as well, although PanOS specifically also seems to require the ability to deal with some XML.

HTTP authentication rituals vary wildly across devices (static tokens, cookies, OAuth, basic auth, ...) and some consideration may be required on what belongs in the input vs. in the model to make this particular input most flexible.

[ytti]

I doubt that the generic use case/support will support the PanOS case. The PanOS case won't be satisfied with model doing HTTP GET for cmd "/directory/file"

[ytti]

If someone gives me access to device with HTTP, I can write the model.

[ghost]

It's hard to give access to a device like that with all the security implication. On my side, I'm backing the Palo Alto devices with a bash script using this method:

curl -s --insecure --request POST 'https://<HOSTNAME>/api/?type=export' --data 'key=<APIKEY>=&category=configuration' | xmllint -format -> $BACKUPDIR/latest_firewall.xml

I don't know if it can help...

[ytti]

You don't have spare Cambium you can put online without any sensitive information?

It's either that or produce the model yourself. :)

[ja-frog]

@ytti I had originally planned on doing it myself but time never seems to allow. Usually I do that kind of thing at night but my little one has been staying up later.

I believe that we have a lab unit...unless someone decided to "borrow" it for production use. I will ask around today and see if management is OK with putting it online then factory-defaulting it afterwards. Paranoia.

How long do you think that you would need it for? Any chance that you will be coming from a specific IP or set of IPs so we could restrict access(shouldn't really matter, but might make some people here happier)?

[ytti]

I would need it for day, but I can't tell which day. Two weeks should be safe. With good luck next weekend will work.

I'd come from 91.198.120.0/24.

[ja-frog]

They opted to throw this off of an ONT in one of our public subnets for subscribers. Password protection. Is there a way to send you the IP, username, and password since GitHub removed its messenger functionality?

[ja-frog]

To retrieve a config via http, the URL is http://DEVICE_IP/canopy_config.cgi

[ytti]

You can use email to saku@ytti.fi, if you want encryption keybase has my key.

[ja-frog]

@ytti First, many thanks for working on this. It works, which is awesome. Because I had issues installing mechanize on the usual box with the CentOS-approved version of ruby installed, I created a new VM where I installed 2.4.2 with mechanize 2.7.5. That meant:

form.CanopyUsername = @node.auth[:username]
form.CanopyPassword = @node.auth[:password]

had to become:

form.field_with(:name => "CanopyUsername").value = @node.auth[:username]
form.field_with(:name => "CanopyPassword").value = @node.auth[:password]

Perhaps a bit off topic, but which version of Ruby and Mechanize should I use?

[millenium7]

I'm going to bump this because i'd like to pull config's from HTTP on all types of Cambium devices, they all handle things a bit differently. ePMP series do not have a fixed URL for the config file (as far as i'm aware). The process with those is to login with username/password, then go to backup/restore section and click on download. This will store a JSON file

The stupid thing with these devices is you can pull the config through SSH but it's in a totally different format, so you can't then just upload that to the web UI. Nor can you easily import those settings

Am happy to provide a lab device to help with the config retrieval

[millenium7]

@ytti I can't get the included cambium.rb example to work either. I don't know what i'm missing or doing wrong

I'm running CentOS7 and have run the following to install the latest version of Python, PIP and Mechanize

sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm
sudo yum update
sudo yum install -y python36u python36u-libs python36u-devel python36u-pip
yum -y install python-pip
pip install mechanize

I added the device in router.db as type cambium along with username and password. I just get 3 failures then it gives up. I've tried this on a number of Cambium PMP devices and they all fail.

What else do I need to do? Here's the full output from Oxidized -d

[oxidized@localhost deviceconfigs]$ oxidized -d
I, [2019-01-23T13:38:37.894878 #16635] INFO -- : Oxidized starting, running as pid 16635
D, [2019-01-23T13:38:37.895911 #16635] DEBUG -- : Hook "email_on_change" registered Exec for event :post_store
I, [2019-01-23T13:38:37.896640 #16635] INFO -- : lib/oxidized/nodes.rb: Loading nodes
D, [2019-01-23T13:38:37.896837 #16635] DEBUG -- : resolving DNS for TestCambiumPMP...
D, [2019-01-23T13:38:37.896895 #16635] DEBUG -- : IPADDR 10.1.0.252
D, [2019-01-23T13:38:37.897012 #16635] DEBUG -- : node.rb: resolving node key 'model', with passed global value of '' and node value 'cambium'
D, [2019-01-23T13:38:37.897134 #16635] DEBUG -- : node.rb: setting node key 'model' to value 'routeros' from global
D, [2019-01-23T13:38:37.897189 #16635] DEBUG -- : node.rb: returning node key 'model' with value 'cambium'
D, [2019-01-23T13:38:37.897232 #16635] DEBUG -- : lib/oxidized/node.rb: Loading model "cambium"
D, [2019-01-23T13:38:37.897701 #16635] DEBUG -- : lib/oxidized/model/model.rb Added #<Proc:0x000000000177e970@/usr/local/rvm/gems/ruby-2.5.1/gems/oxidized-0.24.0/lib/oxidized/model/cambium.rb:2 (lambda)> to the commands list
D, [2019-01-23T13:38:37.897947 #16635] DEBUG -- : node.rb: resolving node key 'input', with passed global value of 'ssh, telnet' and node value ''
D, [2019-01-23T13:38:37.898005 #16635] DEBUG -- : node.rb: returning node key 'input' with value 'ssh, telnet'
D, [2019-01-23T13:38:38.000534 #16635] DEBUG -- : node.rb: resolving node key 'output', with passed global value of 'git' and node value ''
D, [2019-01-23T13:38:38.000646 #16635] DEBUG -- : node.rb: returning node key 'output' with value 'git'
D, [2019-01-23T13:38:38.026843 #16635] DEBUG -- : node.rb: resolving node key 'username', with passed global value of '' and node value '<<>>'
D, [2019-01-23T13:38:38.026963 #16635] DEBUG -- : node.rb: setting node key 'username' to value '<<>>' from global
D, [2019-01-23T13:38:38.027018 #16635] DEBUG -- : node.rb: returning node key 'username' with value '<<>>'
D, [2019-01-23T13:38:38.027079 #16635] DEBUG -- : node.rb: resolving node key 'password', with passed global value of '' and node value '<<>>'
D, [2019-01-23T13:38:38.027128 #16635] DEBUG -- : node.rb: setting node key 'password' to value '<<>>' from global
D, [2019-01-23T13:38:38.027174 #16635] DEBUG -- : node.rb: returning node key 'password' with value '<<>>'
I, [2019-01-23T13:38:38.027293 #16635] INFO -- : lib/oxidized/nodes.rb: Loaded 1 nodes
D, [2019-01-23T13:38:38.393810 #16635] DEBUG -- : lib/oxidized/core.rb: Starting the worker...
D, [2019-01-23T13:38:38.393916 #16635] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 0 of 1
Puma starting in single mode...

• Version 3.12.0 (ruby 2.5.1-p57), codename: Llamas in Pajamas
• Min threads: 0, max threads: 16
• Environment: development
• Listening on tcp://127.0.0.1:8888
  Use Ctrl-C to stop
  D, [2019-01-23T13:38:38.448069 #16635] DEBUG -- : lib/oxidized/worker.rb: Added TestCambiumPMP to the job queue
  D, [2019-01-23T13:38:38.448139 #16635] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
  D, [2019-01-23T13:38:38.448251 #16635] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for TestCambiumPMP at 2019-01-23 02:38:38 UTC
  D, [2019-01-23T13:38:38.448373 #16635] DEBUG -- : lib/oxidized/job.rb: Config fetched for TestCambiumPMP at 2019-01-23 02:38:38 UTC
  W, [2019-01-23T13:38:39.448675 #16635] WARN -- : TestCambiumPMP status fail, retry attempt 1
  D, [2019-01-23T13:38:39.448839 #16635] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 0 of 1
  D, [2019-01-23T13:38:39.449139 #16635] DEBUG -- : lib/oxidized/worker.rb: Added TestCambiumPMP to the job queue
  D, [2019-01-23T13:38:39.449199 #16635] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
  D, [2019-01-23T13:38:39.449612 #16635] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for TestCambiumPMP at 2019-01-23 02:38:39 UTC
  D, [2019-01-23T13:38:39.449713 #16635] DEBUG -- : lib/oxidized/job.rb: Config fetched for TestCambiumPMP at 2019-01-23 02:38:39 UTC
  W, [2019-01-23T13:38:40.449752 #16635] WARN -- : TestCambiumPMP status fail, retry attempt 2
  D, [2019-01-23T13:38:40.449877 #16635] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 0 of 1
  D, [2019-01-23T13:38:40.450022 #16635] DEBUG -- : lib/oxidized/worker.rb: Added TestCambiumPMP to the job queue
  D, [2019-01-23T13:38:40.450105 #16635] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
  D, [2019-01-23T13:38:40.450388 #16635] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for TestCambiumPMP at 2019-01-23 02:38:40 UTC
  D, [2019-01-23T13:38:40.450485 #16635] DEBUG -- : lib/oxidized/job.rb: Config fetched for TestCambiumPMP at 2019-01-23 02:38:40 UTC
  W, [2019-01-23T13:38:41.450562 #16635] WARN -- : TestCambiumPMP status fail, retry attempt 3
  D, [2019-01-23T13:38:41.450709 #16635] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 0 of 1
  D, [2019-01-23T13:38:41.450876 #16635] DEBUG -- : lib/oxidized/worker.rb: Added TestCambiumPMP to the job queue
  D, [2019-01-23T13:38:41.450925 #16635] DEBUG -- : lib/oxidized/worker.rb: 1 jobs running in parallel
  D, [2019-01-23T13:38:41.451224 #16635] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for TestCambiumPMP at 2019-01-23 02:38:41 UTC
  D, [2019-01-23T13:38:41.451324 #16635] DEBUG -- : lib/oxidized/job.rb: Config fetched for TestCambiumPMP at 2019-01-23 02:38:41 UTC
  W, [2019-01-23T13:38:42.451367 #16635] WARN -- : TestCambiumPMP status fail, retries exhausted, giving up
  D, [2019-01-23T13:38:42.451516 #16635] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 1 of 1
  D, [2019-01-23T13:38:42.451593 #16635] DEBUG -- : lib/oxidized/worker.rb: Running :nodes_done hook
  D, [2019-01-23T13:38:43.451960 #16635] DEBUG -- : lib/oxidized/worker.rb: Jobs running: 0 of 1 - ended: 0 of 1

[Packets-n-Python]

Just chiming in as another Palo Alto user. Currently doing a curl GET (G) against the following firewall API with a generated API key, then running it through xmllint and dumping it into the Oxidized configs directory.

curl -skG "https://palofirewall1.foobar.com/api/?type=export&category=configuration&key=<>" | xmllint -format -> /root/.config/oxidized/configs/palofirewall1config.xml