EOS Configs Truncated

[edchiodo]

Running Oxidized with Docker:

Version 3.11.4 (ruby 2.5.1-p57), codename: Love Song
Docker version 19.03.6, build 369ce74a3c

Longer configs for Arista EOS devices are being truncated when writing to disk after a seemingly random number of lines. For instance, when connecting to switch01.domain.net (10.0.0.1) the debug log "10.0.0.1-ssh" shows the complete config with the following commands executed:

switch01#show inventory | no-more
switch01#show running-config | no-more | exclude ! Time:
switch01#exit

However, when comparing the git output to the debug file, we see the output is missing ~450 lines:

root@oxidized:devices.git# cat /opt/oxidized/logs/10.0.0.1-ssh  | wc -l
1439
root@oxidized:devices.git# git show master:switch01.domain.net | wc -l
987

I was originally pointed to a possible prompt desync issue, but after reviewing the configs there is nothing that matches the regex expression /^.+[#>]\s?$/.
Even more odd, where the truncation begins seems to change between Oxidized runs. For instance, switch01.domain.net may have 987 lines backed up on run 1, but on run 2 (4 hours later) only 833 lines are written to output, and on run 3, 1083 lines are written to output...despite no configuration changes being made on the switch.
Currently seeing this behavior on approximately 100 Arista EOS switches running multiple versions of EOS (4.19, 4.20, 4.21). Not seeing similar behavior with IOS, NXOS, MLNXOS, or PANOS devices, which all seem to function correctly.

[edchiodo]

Output of oxidized --show-exhaustive-config is below:

username: username
password: password
model: junos
resolve_dns: true
interval: 14400
use_syslog: false
debug: false
threads: 30
timeout: 200
retries: 2
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
rest: 0.0.0.0:8888
next_adds_job: false
vars:
  auth_methods:
  - none
  - publickey
  - password
  - keyboard-interactive
groups: {}
models: {}
pid: "/root/.config/oxidized/pid"
crash:
  directory: "/root/.config/oxidized/crashes"
  hostnames: false
stats:
  history_size: 10
input:
  default: ssh, telnet
  debug: true
  ssh:
    secure: false
  ftp:
    passive: true
  utf8_encoded: true
output:
  default: git
  git:
    user: oxidized
    email: oxidized@domain.net
    repo: "/root/.config/oxidized/devices.git"
source:
  default: csv
  csv:
    file: "/root/.config/oxidized/router.db"
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      username: 2
      password: 3
      ssh_port: 4
    vars_map:
      enable: 5
    gpg: false
model_map:
  juniper: junos
  cisco: ios
log: "/root/.config/oxidized/logs/log"
hooks:
  push_to_remote:
    type: githubrepo
    events:
    - post_store
    remote_repo: git@github.domain.net:oxidized/oxidized.git
    publickey: "/root/.config/oxidized/.ssh/id_rsa.pub"
    privatekey: "/root/.config/oxidized/.ssh/id_rsa"
  slack:
    type: slackdiff
    events:
    - post_store
    token: <redacted>
    channel: <redacted>

[f0rkz]

I am seeing this as well with EOS devices. It truncates the configuration one minute and then downloads it again the next hour. Our network backup repository is now at 1500 commits. 70% of that is from arista devices just truncating and updating over and over again.

[vincent516]

Unfortunately, we are still having this issue and have been unable to fix it. Saku was working with us on this, but was unable to reproduce the issue when he tried from his server to a test device on our side. If you find a fix, please let me know.

…

On Thu, Jun 3, 2021 at 10:26 AM Nick ***@***.***> wrote: I am seeing this as well with EOS devices. It truncates the configuration one minute and then downloads it again the next hour. Our network backup repository is now at 1500 commits. 70% of that is from arista devices just truncating and updating over and over again. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2038 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOHJ7HJPI7JRBOWC2KVKC6LTQ6GJNANCNFSM4K4N4C3A> .

[f0rkz]

Unfortunately, no love here... Anyone able to find a fix for this?

@vincent516 perchance are you using 7050's?

[vincent516]

We are not using 7050. We have a mix of Arista models and it happens across all of them unfortunately.

[Justin417]

Hello!

We're experiencing the same issue. Our environment contains dozens of switches, about half being Arista and half being Juniper. Out of all these hosts, only 3 or so Arista switches experience this issue.

Like @edchiodo saw, our SSH debug output shows the full configuration file successfully being retrieve over SSH, ruling out an SSH-specific issue. We see several hundred lines missing between the raw SSH data and what oxidized outputs:

# git show :lf220a.pit1.teraswitch.com | wc -l
1720
# cat 10.100.10.121-ssh | wc -l
2856

We're unable to find the root cause of this, my colleagues and I thought perhaps some rogue escape character or malformed character was located in the file causing Oxidized to cut the config short, but looking at where the Oxidized cutoff occurs, there are no non-standard characters, and the location of the cutoff appears completely random.

Has anyone been able to find a resolution at this time?

@f0rkz If it matters at all, we're seeing this issue exclusively on 7050s at this time.

[vincent516]

We still haven't been able to find a resolution.

…

On Thu, Dec 9, 2021 at 4:45 PM Justin Goetz ***@***.***> wrote: Hello! We're experiencing the same issue. Our environment contains dozens of switches, about half being Arista and half being Juniper. Out of all these hosts, only 3 or so Arista switches experience this issue. Like @edchiodo <https://github.com/edchiodo> saw, our SSH debug output shows the full configuration file successfully being retrieve over SSH, ruling out an SSH-specific issue. We see several hundred lines missing between the raw SSH data and what oxidized outputs: # git show :lf220a.pit1.teraswitch.com | wc -l 1720 # cat 10.100.10.121-ssh | wc -l 2856 We're unable to find the root cause of this, my colleagues and I thought perhaps some rogue escape character or malformed character was located in the file causing Oxidized to cut the config short, but looking at where the Oxidized cutoff occurs, there are no non-standard characters, and the location of the cutoff appears completely random. Has anyone been able to find a resolution at this time? @f0rkz <https://github.com/f0rkz> If it matters at all, we're seeing this issue exclusively on 7050s at this time. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2038 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOHJ7HKLHEOHWZBGONPENUTUQEPOXANCNFSM4K4N4C3A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jake2184]

I've been having this issue, and then after digging into it I think I've fixed it by tweaking the prompt in lib/oxidized/model/eos.rb.

Current:

prompt /^.+[#>]\s?$/

Fixes the issue for me (On DCS-7050):

prompt /^.+[#>]$/

Configs are now being pulled consistently in full.

I presume the \s? is there for a reason for other Arista models? I assume a new/extended model is needed rather than editing the plain eos.rb one?

[Justin417]

Huge thanks to @jake2184 for the find!

I can confirm, we applied the fix to our environment and have not seen the issue re-occur for several days now on any DCS-7050 in our inventory at this time.

[mortzu]

Fixed in #2489