Could not generate persistent MAC address for s-010255213041: No such file or directory

[Houlistonm]

Could not generate persistent MAC address for s-010255213041: No such file or directory

Summary

Observed errors in logs on the diego-cell vms.

Steps to Reproduce

Install diego-cell vm (these errors were found in the toolsmiths environment)

Diego repo

Environment Details

Name  Release(s)                    Stemcell(s)                                     Team(s)  
cf    binary-buildpack/1.0.42       bosh-google-kvm-ubuntu-xenial-go_agent/621.125  -  
      bosh-dns/1.27.0                                                                 
      bosh-dns-aliases/0.0.4                                                          
      bpm/1.1.16                                                                      
      capi/1.127.0                                                                    
      cf-cli/1.38.0                                                                   
      cf-networking/2.43.0                                                            
      cf-smoke-tests/41.0.2                                                           
      cflinuxfs3/0.276.0                                                              
      credhub/2.12.0                                                                  
      diego/2.61.0                                                                    
      dotnet-core-buildpack/2.3.39                                                    
      garden-runc/1.20.1                                                              
      go-buildpack/1.9.39                                                             
      java-buildpack/4.48.1                                                           
      log-cache/2.11.6                                                                
      loggregator/106.6.2                                                             
      loggregator-agent/6.3.8                                                         
      metrics-discovery/3.0.8                                                         
      nats/43                                                                         
      nginx-buildpack/1.1.35                                                          
      nodejs-buildpack/1.7.67                                                         
      php-buildpack/4.4.56                                                            
      pxc/0.41.0                                                                      
      python-buildpack/1.7.49                                                         
      r-buildpack/1.1.26                                                              
      routing/0.231.0                                                                 
      ruby-buildpack/1.8.51                                                           
      silk/2.43.0                                                                     
      staticfile-buildpack/1.5.29                                                     
      statsd-injector/1.11.18                                                         
      uaa/75.15.0                                                  

Possible Causes or Fixes (optional)

Discovered/observed these errors during a cf-networking-onboarding taught to developers at vmware.
Googled for the error and found this old/open systemd issue

The discussion mentions editing a file /usr/lib/systemd/network/99-default.link and changing MACAddressPolicy=persistent to MACAddressPolicy=none

The diego cells have four instances of that file:

/var/vcap/data/grootfs/store/unprivileged/images/35746888-82c4-483f-5b28-1953/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/images/35746888-82c4-483f-5b28-1953-envoy/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/images/35746888-82c4-483f-5b28-1953-liveness-healthcheck-0/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/volumes/1fda073f8c13d49c63ba8a222e867dab99f7bddc859bb792f4aff6f41b19d146/lib/systemd/network/99-default.link

Made the change to the healthcheck file, restart bosh-dns-healthcheck and the errors that previously occurred every 20 seconds have abated.
A lower frequency error persists ... from one of the other process I'm sure.

Additional Text Output, Screenshots, contextual information (optional)

[geofffranks]

Hey @Houlistonm! I'm taking a look at this issue, and have a few questions:

1. When you encountered the problem, what impact was seen on the diego cell? Were there issues with containers/applications?
2. What logs did this error occur in?
3. Your notes indicate that the issue is related to systemd files on the container rootfs for applications, but you also indicate modifying a healthcheck file, and restarting the bosh-dns-healthcheck job via monit to resolve the issue. Can you elaborate on which file you modified, and which errors occurred every 20s? Did restarting the application or rep job have any effect?
4. Are the reproduction simply the following?
   i. Deploy CF with diego-release 2.61.0, cflinuxfs3 0.276.0, and the ubuntu-xenial 621.125 stemcell
   ii. Push an app
   iii. Review a log somewhere on the diego-cell.

[Houlistonm]

I no longer have access to toolsmith environment ... so I'm answering your questions based on my TAS 2.13.7 (latest) foundation.

1. I couldn't discern any impact to the diego cells/containers/apps - appears to be a noisy error
2. # grep systemd-udev /var/log/syslog

2022-07-25T16:20:00.394030+00:00 073351a7-289f-427d-a195-dbb47a3f26a5 systemd-udevd[2846270]: Could not generate persistent MAC address for c-010255073211: No such file or directory
2022-07-25T16:20:00.394423+00:00 073351a7-289f-427d-a195-dbb47a3f26a5 systemd-udevd[2846271]: Could not generate persistent MAC address for s-010255073211: No such file or directory
2022-07-25T16:30:01.035018+00:00 073351a7-289f-427d-a195-dbb47a3f26a5 systemd-udevd[2865283]: Could not generate persistent MAC address for c-010255073212: No such file or directory
2022-07-25T16:30:01.047744+00:00 073351a7-289f-427d-a195-dbb47a3f26a5 systemd-udevd[2865284]: Could not generate persistent MAC address for s-010255073212: No such file or directory
2022-07-25T16:50:02.385329+00:00 073351a7-289f-427d-a195-dbb47a3f26a5 systemd-udevd[2902798]: Could not generate persistent MAC address for c-010255073214: No such file or directory
2022-07-25T17:00:03.076554+00:00 073351a7-289f-427d-a195-dbb47a3f26a5 systemd-udevd[2921830]: Could not generate persistent MAC address for s-010255073215: No such file or directory

3. # find /var/vcap/data -type f -name 99-default.link | sort -u

/var/vcap/data/grootfs/store/unprivileged/images/0e6d3ce4-cef8-4999-5b3e-f540-envoy/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/images/0e6d3ce4-cef8-4999-5b3e-f540-liveness-healthcheck-0/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/images/0e6d3ce4-cef8-4999-5b3e-f540/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/images/1b6b6523-9031-4be0-5977-c532-envoy/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/images/1b6b6523-9031-4be0-5977-c532-liveness-healthcheck-0/rootfs/lib/systemd/network/99-default.link
... <REDACTED FOR BREVITY>
/var/vcap/data/grootfs/store/unprivileged/images/f2cd1c98-df7d-4482-6466-5322-envoy/rootfs/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/volumes/90ee055d3d666f6b1c947802e5b5ca99c45ad0867afbb5e6bfb67c6e4218db9c/lib/systemd/network/99-default.link
/var/vcap/data/grootfs/store/unprivileged/volumes/f42e01ab88f6bd13845157f446679d1da2543af9bc168938c0f2900926f2fd04/lib/systemd/network/99-default.link

It appears I created the file /etc/systemd/network/99-default.link with the contents:

[Link]
MACAddressPolicy=none

This was a crude/brute-force attempt to affect the frequency of the error ... I'm sure there is a more elegant solution.
4. I found these logs on my TAS 2.13.7 foundation ... so it appears any running diego-cell vm should provided an environment to investigate the error.

[geofffranks]

Good news, I was able to confirm the error was there as of silk-release v3.9.0. After updating to silk-release v3.10.0, the error is no longer present, due to changes in how we are creating the container interfaces. This should be available in the next cf-deployment release.

Going to close this out as fixed!