astats providing bad interface data with virtio KVM driver on ATS servers

[jhowell-comcast]

When using the virtio driver on a KVM / QEMU VM running ATS, astats reports the interface speed (inf.speed) as zero. This causes Traffic Monitor to do odd things and report available bandwidth as a negative number. This doesn't allow cache groups to come online and Traffic Router attempts to do DS_BYPASS as it thinks the ATS servers are maxed out on bandwidth.

Using the e1000 driver in KVM / QEMU VM provides expected behaviors.

[rawlinp]

👍 as a super hack workaround, you might be able to configure the minimum available bandwidth parameter in the TM profile as a negative number to "trick" TM into thinking there is enough bandwidth available. I believe that was something I had to do when running ATS on a VM using the virtio driver.

Virtio reports the NIC speed as zero, and TM doesn't know how to handle that. I'm not sure how it should handle that (ignore bandwidth availability? give it a default NIC speed?).

[jhowell-comcast]

I tried the super hack workaround, it did enable TM to online the cache groups, but TR still tried to do DS_BYPASS.

This is totally from running ATS on a KVM / QEMU VM with virtio driver. The virtio driver doesn't report zero for the interface speed. It reports nothing. astats is assigning a zero value to the variable. Perhaps a default nic speed if it's virtio that won't trigger tm to shut it down.

It is easily gotten around by using e1000 driver in the VM, but you lose some performance from that.

This isn't likely something that will ever be bumped into in a production instance, but somewhat likely in lab.

[jhg03a]

If we could have a "default_bw" file or ats profile parameter, that could be used in the case where normal bandwidth detection methods can't function.

[jhg03a]

Another option is to add a new astats.config parameter for this as well.

[jhg03a]

Upon further thought it seems like either it's going to be a fix in astats with a new default_link_speed parameter in astats.config, or we simply remove bandwidth calculation from the TM health protocol in the case where the link speed cannot be determined.

The reason virtio doesn't have a link speed is because it's not a concept with that network driver. It's simply shoving bits at the hypervisor as fast as possible which means sometimes you may have exceptionally fast transfers while other times they may be congested. The e1000 driver still has the same problems but it establishes an upper bound to emulate physical hardware with a 1G interface instead of taking advantage of a deeper integration with the hypervisor.

[rawlinp]

I'm leaning towards the profile parameter option just because using a VM with the virtio driver should be a very targeted use case. Astats should just return the default bandwidth if the link speed cannot be determined or is < 1.

[jhg03a]

I found a workaround in Openstack by adding an additional piece of metadata to the base image for now that'd gotten me unblocked for now, but may not be consistent/available in all Openstack implementations.