Tune iscsid.conf for better performance

[zedi-pramodh]

During a POC with a customer this regression was found that queue depth was set to 1 in initial development phase of eve-k and never fixed back.

So set it SSD devices, which we expect as a minimum spec for clustering nodes.

HDD spinning disks:
queue_depth = 64 ← HDDs can't handle deep queues well
cmds_max = 128

SSD (SATA):
queue_depth = 128 ← SSDs handle deeper queues well
cmds_max = 256

NVMe:
queue_depth = 256 ← NVMe has native deep queue support
cmds_max = 512

How to test and validate this PR

1. Create 3 node cluster
2. Deploy multiple VMs.
3. Run some activity in the VMs and watch the performance.
4. Do few failovers and see how fast the replicas are syncing.

You should see response times improved before and after this PR

Changelog notes

None

Checklist

• I've provided a proper description

• I've added the proper documentation

• I've tested my PR on amd64 device

• I've written the test verification instructions

• I've set the proper labels to this PR

• I've checked the boxes above, or I've provided a good reason why I didn't
  check them.

Please, check the boxes above after submitting the PR in interactive mode.

[zedi-pramodh]

@andrewd-zededa can you review this one.

[zedi-pramodh]

@andrewd-zededa can you look into his PR.

[andrewd-zededa]

@zedi-pramodh my only worry here is changing the default for all systems to unlock the queue depth for the user/app volumes. For lower resource systems where k3s db shares the same disk and fs this may enable the ability to starve out k3s for io performance. Can the default be set but configured as an eve config property so that other systems can be debugged without a new eve build? I would propose having zedkube:

1. read the config property
2. set this as a value in types.KubeConfig
3. Then the kube service container can read the value and overwrite iscsid.conf

[zedi-pramodh]

@zedi-pramodh my only worry here is changing the default for all systems to unlock the queue depth for the user/app volumes. For lower resource systems where k3s db shares the same disk and fs this may enable the ability to starve out k3s for io performance. Can the default be set but configured as an eve config property so that other systems can be debugged without a new eve build? I would propose having zedkube:

1. read the config property
2. set this as a value in types.KubeConfig
3. Then the kube service container can read the value and overwrite iscsid.conf

I am not clear on your suggestion. The default values I set are actually recommended as default for SSD drives. Even if we change through eve config are we saying we have to restart all the apps or reboot nodes for this config change ?
Also leaving this setup and expecting customers to tune is bit overhead in IMO.

[andrewd-zededa]

@zedi-pramodh my only worry here is changing the default for all systems to unlock the queue depth for the user/app volumes. For lower resource systems where k3s db shares the same disk and fs this may enable the ability to starve out k3s for io performance. Can the default be set but configured as an eve config property so that other systems can be debugged without a new eve build? I would propose having zedkube:

1. read the config property
2. set this as a value in types.KubeConfig
3. Then the kube service container can read the value and overwrite iscsid.conf

I am not clear on your suggestion. The default values I set are actually recommended as default for SSD drives. Even if we change through eve config are we saying we have to restart all the apps or reboot nodes for this config change ? Also leaving this setup and expecting customers to tune is bit overhead in IMO.

The intended user of the config property would be for dev/debug on new systems, not end-user operators.

[zedi-pramodh]

@zedi-pramodh my only worry here is changing the default for all systems to unlock the queue depth for the user/app volumes. For lower resource systems where k3s db shares the same disk and fs this may enable the ability to starve out k3s for io performance. Can the default be set but configured as an eve config property so that other systems can be debugged without a new eve build? I would propose having zedkube:

1. read the config property
2. set this as a value in types.KubeConfig
3. Then the kube service container can read the value and overwrite iscsid.conf

I am not clear on your suggestion. The default values I set are actually recommended as default for SSD drives. Even if we change through eve config are we saying we have to restart all the apps or reboot nodes for this config change ? Also leaving this setup and expecting customers to tune is bit overhead in IMO.

The intended user of the config property would be for dev/debug on new systems, not end-user operators.

For that we can modify locally right ?
iscsiadm -m node -o update -n -v

[andrewd-zededa]

@zedi-pramodh my only worry here is changing the default for all systems to unlock the queue depth for the user/app volumes. For lower resource systems where k3s db shares the same disk and fs this may enable the ability to starve out k3s for io performance. Can the default be set but configured as an eve config property so that other systems can be debugged without a new eve build? I would propose having zedkube:

1. read the config property
2. set this as a value in types.KubeConfig
3. Then the kube service container can read the value and overwrite iscsid.conf

I am not clear on your suggestion. The default values I set are actually recommended as default for SSD drives. Even if we change through eve config are we saying we have to restart all the apps or reboot nodes for this config change ? Also leaving this setup and expecting customers to tune is bit overhead in IMO.

The intended user of the config property would be for dev/debug on new systems, not end-user operators.

For that we can modify locally right ? iscsiadm -m node -o update -n -v

Ok that can work for non-persistent testing. I think long term there should be some facility to configure settings from the controller and persist them.

[zedi-pramodh]

changed values after testing on slower disks. Please check.

[eriknordmark]

LGTM

[zedi-pramodh]

@zedi-pramodh my only worry here is changing the default for all systems to unlock the queue depth for the user/app volumes. For lower resource systems where k3s db shares the same disk and fs this may enable the ability to starve out k3s for io performance. Can the default be set but configured as an eve config property so that other systems can be debugged without a new eve build? I would propose having zedkube:

1. read the config property
2. set this as a value in types.KubeConfig
3. Then the kube service container can read the value and overwrite iscsid.conf

I am not clear on your suggestion. The default values I set are actually recommended as default for SSD drives. Even if we change through eve config are we saying we have to restart all the apps or reboot nodes for this config change ? Also leaving this setup and expecting customers to tune is bit overhead in IMO.

The intended user of the config property would be for dev/debug on new systems, not end-user operators.

For that we can modify locally right ? iscsiadm -m node -o update -n -v

Ok that can work for non-persistent testing. I think long term there should be some facility to configure settings from the controller and persist them.

I agree for longterm we should look at config value and tune test it on different HW systems, may be we can provide to HW certification team.