Cisco NX-OS: 2-5% of Devices Fail ZTP Boot Due to Incorrect Password String Generation

[swagner-de]

Description

During Zero Touch Provisioning (ZTP) boot, 2-5% of Cisco NX-OS devices fail to provision with valid admin credentials, requiring manual password reset and preventing scalable automation.

Credentials Delivery during ZTP

We provision users by sending only scrypt-encrypted, salted password hashes (never plaintext) to prevent credential leakage:

1. Generate random 10-byte salt using crypto/rand
2. Create scrypt hash (N=16384, r=8, p=1, keyLen=32)
3. Build NX-OS password string using Cisco's undocumented custom base64 alphabet: ./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
4. Format as: $nx-scrypt$<base64_salt>$<base64_key>
5. Apply as username $user password 9 $passwordString role network-admin to the scheduled-config, the configuration that is replayed after the POAP boot has finished.

Suspected Issue

Incorrect base64 alphabet or other hash failures Since Cisco provides no official documentation on type 9 password generation, we reverse-engineered the format. The custom base64 alphabet was sourced from some corner of the internet and may be incorrect.

Reproduction

To confirm our suspicions I had claude generate a little test tool . This tool creates different credentials, creates them as different users on the device and tries them out. If I create the all users with the same password, just with different salts, I reach around 1-10 credentials that do not work. I think this pretty much confirms our suspicion that something breaks while generating the password string.

The tool is self-contained an can be run against any Cisco NXOS device using
go run main.go -configure -count 100 -password my-cisco-test123 -host 1.1.1.1 -admin-user admin -admin-pass Cisco123

An example result can be seen here.

Vendor Engagement

We are currently seeking vendor support to get official information on the password string generation.

Workarounds

1. Use pre-validated salt+password combinations (high risk of leaking credentials)
2. Disable ZTP (current approach - blocks automation)
3. Use static ZTP password, rotate post-boot (security risk window)

ZTP disabled until resolution. If vendor support is slow, will implement workaround 3.

[swagner-de]

Upon answer from Cisco we got to know that zero-bytes in the salt are causing this condition. We tried it out with around 2000 salts/users and do not see this condition any more.
Probability also checks out. 1/256 chance of a byte being zeroed, x 10 bytes =~ 4%.