Clarifiction on Multiple Deployments notes #210
Comments
|
These are the TEST PVCs: Respective TEST PVs: Example of the TEST PVC from Prometheus: DEV PVCs: Hours later, still no signs of disk usage on TEST: DEV space is being consumed: Also the two clusters are fighting over the same PVs. I have observed multiple times where the dashboard in TEST stops working, to even showing no dashboards available.... but then DEV starts working for a while. And they flip flop now and then. so when its not winning, I see restarts in the iscsi csi-driver logs: |
|
Welcome! Your understanding is firm on how things work and what the ramifications are. I’m guessing nfs is working fine and so the issue is isolated only to iscsi. The issue with iscsi is that the target/extent names are in a ‘global’ namespace because you’re not actually strictly speaking at that layer using zfs semantics. To alleviate this issue see lines 72-74 around here: https://github.com/democratic-csi/democratic-csi/blob/master/examples/freenas-iscsi.yaml#L73 Review the notes there and be cognizant of the limits, and then I would set the prefix or suffix to be your cluster identifier (ie: dev or test) and you shouldn’t see fighting any longer. Be sure to delete all the current assets and start clean before starting your test as well. I would delete in k8s and then also manually ensure on the storage system that assets look sane as you may have some weirdness based on what has transpired to this point. |
|
Actually unless the pvs have the same name that would be pretty unlikely to collide as well. Can you send over the pv yaml from both clusters (remove any sensitive info)? That may help me pinpoint what could be going on. |
|
I was already uninstalling and cleaning up. Completed a full cleanup on both clusters. CSI cleanup did not need any manual work, everything deleted on its own: Uninstalled CSI iSCSI Helm as well. Now I have Ansible render the Now the iSCSI target and extent names, look different:
I let ArgoCD do its magic to redeploy the deltas and all ZVOls started to show growth: I'll let things run overnight and see if any driver pods got restarted. |
|
Do you have |
|
I do not. it's whatever the default is. No restarts on the driver pods overnight. Everything seems stable. Do you still want the pv yaml ? |
|
Yeah and send over a fully rendered config too. I think the template must be set because normally the iscsi assets have the pvc id in there. Using namespace/name is helpful but will cause issues if the lengths of those grow (allowed lengths in k8s will exceed the allowed lengths of the iscsi names). |
|
Prometheus DB PV: Prometheus AlertManager PV: Grafana PV: |
|
I pulled this from ArgoCD showing the |
|
The nameTemplate seems to be the default value in the secret: |
|
Yeah that is an example, not the default. Just comment that line altogether for anything even remotely production. |
|
Delete all pvs again and start over fyi. |
|
Done. The names of the Targets and Extents are much smaller now, each looks something like: Such as: Instead of: So where I think I went wrong originally was to uncomment the My fault for the assumption about I would recommend you include in the I assume this is good now? |
|
Yeah agreed on the additional note. Thanks for bringing it up and working through it! Let’s leave it open until the doc is updated. |
|
Added some extra notes in the README and merged. |
|
Typo? does seem to be a link to #210: |
|
Yeah I’ll see if I can get the syntax correct. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The main
README.mdhas the following comments:I have two independent K3s clusters with nothing shared but TrueNAS backend for iSCSI and NFS. Two independent ArgoCD repositories are used to build each. I'll call them
DEVandTEST. Essentially deployment file are identical between them other than ingress route names, IP addresses, etc. mostly minor items. Ansible is used to build each of them.Regarding Democratic-CSI each cluster points to a different ZFS parent dataset but otherwise they are the same deployment. There is only ONE deployment of ISCSI provider and NFS provider per cluster. All other TrueNAS connectivity between the two is the same same user ID for SSH, same iSCSI connection configuration.
What I observed on a deployment of Kube Prometheus Stack to DEV - flawless, everything worked as expected. no issues. Created 3 iSCSI PV in the correct parent dataset,
I then went to deploy it to TEST, and it got a little weird:
Based on that I assumed all was OK. However, when I log into TEST ENV Grafana which worked, the data is weird. I see my Work in Progress dashboards from the DEV environment that have yet to be deployed to TEST. These dashboards were not part of the ArgoCD deployment for TEST, its somehow picking them up from the database.
I check on TrueNAS, the status of the 3 ZVOLs created for TEST which are in Bond status... and each of them are still just 88Kib in size and not growing. They are clearly not being used.
Both clusters generated the same claim names (expected), with same StorageClass, Reclaim Policy, Access Mode, Capacity. 6 PVs got created, each with different volume name, each of the volume names line up to a ZVOL in the correct parent dataset and all six report "bound" status. It seems like everything got created correctly, but at some layer the CSI is confused.
I'm wondering if its
Make sure you have a unique csiDriver.name in the values fileis that unique per cluster? or need to be unique across all clusters?The text was updated successfully, but these errors were encountered: