Fix create method to handle error

[mikewiebe]

This fixes the following error in the dns_domain minitest when run on an n6k.

  1) Error:
TestDnsDomain#test_dnsdomain_create_destroy_multiple_vrf:
Cisco::CliError: CliError: 'vrf context test ;  ip domain-list aoeu.com' rejected with message:
'ERROR: Deletion of VRF test in progresswait for it to complete'
    /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/lib/cisco_node_utils/node.rb:266:in `rescue in config'
    /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/lib/cisco_node_utils/node.rb:263:in `config'
    /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/lib/cisco_node_utils/node.rb:161:in `config_set'
    /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/lib/cisco_node_utils/node_util.rb:58:in `config_set'
    /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/lib/cisco_node_utils/dns_domain.rb:68:in `create'
    /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/lib/cisco_node_utils/dns_domain.rb:37:in `initialize'
    tests/test_dns_domain.rb:100:in `new'
    tests/test_dns_domain.rb:100:in `test_dnsdomain_create_destroy_multiple_vrf'

MiniTest Results

Node under test:
  - name  - n6k-92
  - type  - N6K-C6001-64P
  - image - bootflash:///n6000-uk9-kickstart.7.3.0.N1.0.270.bin

TestDnsDomain#test_dnsdomain_create_destroy_multiple = 10.27 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single_vrf = 4.67 s = .
TestDnsDomain#test_dnsdomain_create_destroy_multiple_vrf = 11.61 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single = 4.29 s = .

Finished in 30.841510s, 0.1297 runs/s, 1.6212 assertions/s.

4 runs, 50 assertions, 0 failures, 0 errors, 0 skips

rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> ruby tests/test_dns_domain.rb -v -- n7k-j admin admin
Run options: -v -- --seed 40650

# Running:


Node under test:
  - name  - n7k-j
  - type  - N7K-C7009
  - image - bootflash:///n7000-s2-dk9.7.3.0.D1.0.202.bin

TestDnsDomain#test_dnsdomain_create_destroy_multiple_vrf = 4.72 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single = 2.73 s = .
TestDnsDomain#test_dnsdomain_create_destroy_multiple = 3.43 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single_vrf = 2.93 s = .

Finished in 13.819536s, 0.2894 runs/s, 3.6181 assertions/s.

4 runs, 50 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest to /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/coverage. 595 / 1193 LOC (49.87%) covered.
rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> 
rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> ruby tests/test_dns_domain.rb -v -- n8k-136 admin admin
Run options: -v -- --seed 52839

# Running:


Node under test:
  - name  - n8k-136
  - type  - N5K-C56128P
  - image - bootflash:///n6000-uk9-kickstart.7.3.0.N1.0.281.bin

TestDnsDomain#test_dnsdomain_create_destroy_multiple_vrf = 11.49 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single_vrf = 8.94 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single = 7.50 s = .
TestDnsDomain#test_dnsdomain_create_destroy_multiple = 9.72 s = .

Finished in 37.649872s, 0.1062 runs/s, 1.3280 assertions/s.

4 runs, 50 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest to /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/coverage. 607 / 1193 LOC (50.88%) covered.
rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> 
rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> ruby tests/test_dns_domain.rb -v -- dt-n9k5-1 admin admin
Run options: -v -- --seed 60044

# Running:


Node under test:
  - name  - dt-n9k5-1
  - type  - N9K-C9396PX
  - image - bootflash:///nxos.7.0.3.I3.0.278.bin

TestDnsDomain#test_dnsdomain_create_destroy_single = 3.98 s = .
TestDnsDomain#test_dnsdomain_create_destroy_multiple = 3.29 s = .
TestDnsDomain#test_dnsdomain_create_destroy_multiple_vrf = 3.53 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single_vrf = 3.15 s = .

Finished in 13.953775s, 0.2867 runs/s, 3.5833 assertions/s.

4 runs, 50 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest to /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/coverage. 598 / 1193 LOC (50.13%) covered.
rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> 
rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> ruby tests/test_dns_domain.rb -v -- n3k-105 admin admin
Run options: -v -- --seed 16828

# Running:


Node under test:
  - name  - n3k-105
  - type  - N3K-C3048TP-1GE
  - image - bootflash:///nxos.7.0.3.I3.0.285.bin

TestDnsDomain#test_dnsdomain_create_destroy_multiple_vrf = 9.32 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single_vrf = 4.56 s = .
TestDnsDomain#test_dnsdomain_create_destroy_single = 4.20 s = .
TestDnsDomain#test_dnsdomain_create_destroy_multiple = 5.43 s = .

Finished in 23.514448s, 0.1701 runs/s, 2.1264 assertions/s.

4 runs, 50 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for MiniTest to /nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils/coverage. 598 / 1193 LOC (50.13%) covered.
rtp-ads-432:/nobackup/mwiebe/REPOS_NODE_UTILS/release120/cisco-network-node-utils> 

[glennmatthews]

Looks like you're retrying as fast as possible. Would it make more sense to retry up to a max of, say, 30 times, but sleep 1 second between each try?

How long does 500 retries take? What's the maximum number of retries you've seen before it succeeds?

[mikewiebe]

Was hoping you would bring that up :). Yes, I am trying as fast as possible and wanted your feedback. It takes around 120 attempts which is just around 1-3 seconds. Very fast. I hesitated to add a sleep because I wanted it to bail out as quickly as possible but I am fine with a 1 second sleep. Thoughts?

[glennmatthews]

I'd definitely prefer a sleep of 1 s (or even 0.5 s if that's straightforward to do). If the slowest you've seen is 3 seconds, I'd rather retry 10 times with a 0.5s delay (or 5 times with a 1s delay) than 120 times in the same time period.

[mikewiebe]

@glennmatthews 10 tries with a 1 second sleep seems to be the best. My testing resulted in an average of 6 attempts with this setting.