Slow server responses causing PV attachment timeouts #309
Comments
|
Here's some benchmarks with 679 zfs datasets ZFS version: The rest of the commands in |
|
I could try passing through the vol and snapshot data here: Then construct the I will need to figure out how to build the linstor-server and dockerize it (using https://github.com/piraeusdatastore/piraeus/tree/master/dockerfiles/piraeus-server) into the piraeus-server image (I can host it on my own repo for testing). |
|
I am not sure if I understood you correctly, but if you mean that Linstor might optimize some |
|
@ghernadi do you have any load? The slowness of As stated here: It may be better to move away from the ZFS CLI entirely as it doesn't look like there's a fix for this, if a user had many datasets in their Linstor pool they will come across this issue eventually. In the meantime I will still work to see if listing the datasets individually will improve performance (it definitely will in our case). Here's a list of issues about the slowness of openzfs/zfs#8898 |
No, and from the links you shared I do believe you that Feel free to make suggestions on how to improve Linstor or make it less dependent on |
|
@ghernadi Can you take a look at Rid@427185c it's just a quick idea which could speed up systems where there are many datasets outside of linstor (such as our case). |
|
I'd like to be able to build deb packages for testing however I can't find the tooling for doing the build, I can see the debian folder but no DEBIAN folder. Can you let me know how to build the .debs? |
|
I ended up putting the built source into a docker container, however running satellites is generating errors and not connecting: Do the servers & satellites all need to be the same version? At the moment I only have the satellites running this code. |
|
The previous error was resolved by updating the server containers to the same version. However the I'll take another look at it and see if I can work around. |
|
Ok I've tested the following commit Rid@458216a Before the changes: After the changes: Would you like me to submit a PR? |
|
Actually I'm thinking we could make this a lot more efficient if instead of looping through every volume, we just group them by storage pool and then list the storage pools. I'll adjust the commit. |
|
Here is the commit for only listing unique storage pools Rid@7ccceac It doesn't actually make any noticeable different from the previous commit, but it should do if you have pools with more datasets. I'm not sure what the bottleneck is in this case which is pushing up the response to 700ms in the linstor command. Looks like some of it is creating the json object. I think 300ms should eliminate the timeout problem. |
Good idea, but instead of looping over all storage pools and calling individual
Something like But I assume you already figured that out :)
Yes, but that does not seem to be the problem of the ErrorReport. After the controller established the connection to the satellite, the satellite reports its own Linstor version along with other things. If the controller has a different version than the satellite, the controller closes the connection again and marks the satellite as |
|
That makes sense, I will test it now and report back. |
|
@ghernadi tests were successful, performance is fine, I think it can be merged. The actual Do you have any idea where the extra 350ms could be coming from on The |
|
I might be wrong here, but there might be quite a delay in opening a socket to the controller or something like that. I'd have to investigate to refresh my memories, but I believe about 200-300ms are lost between client <-> controller. Maybe @rp- knows more in this regard |
|
|
There's only 2 satellites, which takes around 20ms on each to do a zfs list, but shouldn't these requests be asynchronous? In which case we'd expect all values to be returned by 20ms. Worst case and it's not async would be 40ms. The servers are both in the same rack, so RTT should be close to 0-5ms. |
We're using piraeusdatastore to provide linstor PVs to a k8s cluster which we're currently building.
We're testing various scenarios with 2 worker nodes and a tiebreaker where we want automatic fail-over in case of a single node failure.
We initially noticed that upon node failure where the PVs were attached to pods, the PV attachments on the non-failing node (upon fail-over) were timing out and ultimately never completed (the pods were stuck in pending state):
We submitted an issue over at the piraeusdatastore repo here piraeusdatastore/linstor-csi#172.
We then did some pruning on the system deleting old images and stale containers. We're running ZFS on root, so each container and image is using a zfs dataset.
After the pruning the PVs would attach on the non-failing node after 15 mins.
When the PVs were not attaching we had around 1300 datasets per node when doing
zfs list | wc -l, after the pruning this got down to 679. However in production we expect this number to be higher than 1300.With 679 zfs datasets
zfs listtakes about 1.6-1.8 secs. Conversely if we dozfs list -ron the pool assigned to linstor it only takes 14ms. Would it be much faster to list all of the pools attached to linstor individually?You can see the exact requests which are timing out on the linstor-server in this reply on the issue piraeusdatastore/linstor-csi#172 (comment)
Listing resources is also slow (unsure if this is normal):
The text was updated successfully, but these errors were encountered: