-
Notifications
You must be signed in to change notification settings - Fork 66
Jenkins is not unidled for 1a cluster user #4378
Comments
There is no "jenkins-home" PVC under that namespace, which explains why the jenkins pod refuses to start. |
downgradding to SEV2 since that can only occur for a subset of users in sev1 |
Who is responsible for creating this PVC? |
@ppitonak it must be created as a part of I am curious to know if we have any logs about tenant initialisation for this namespace. |
@jfchevrette I don't think it's solely because of that... LOGS:
Nothing more is added to the log, until the deployment times out, because readinnes probe didn't succeed. |
Thanks for the investigation @rhopp, @jfchevrette where can we look to see the gluster IO sisue ? this is IO or networking who would shows this, |
corresponds to https://github.com/fabric8-jenkins/jenkins-openshift-base/blob/5ff927d8786e9e43ed629236e932fe034e93958d/2/contrib/s2i/run#L308 |
Gluster monitoring in zabbix: |
@chmouel why did you decrease severity? See the definition of SEV-1 from https://access.redhat.com/support/policy/severity
For at least three users the production system is not functioning and no workaround exists |
@ppitonak I assume it's because it's only affects 1a and not everyone, I don't mind really having it in SEV-1 if you want it as is :) |
@chmouel if you refer to the problem with volumes with more than 20k files this is the PR that @benoitf has created. The fix should be available in 3.11. I don't know when it will be available on OSO honestly. |
It seems that there are multiple issues here... should I log |
@ppitonak good idea, |
@l0rd i guess that's would improve the volume mounting slowness but unfortunately not the slow IO writes on gluster, |
@jmelis I am assigning this to you (need a point of contact) since does not seems like an idler or a jenkins issue but some I/O bound problem . |
May be related #4121 |
I see the following event in OpenShift console for account
BTW, is this resource quota shared with Che workspaces? |
The original issue was about jenkins not coming up in the Ref. https://bugzilla.redhat.com/show_bug.cgi?id=1631834 The reason why the If there are other issues observed in other namespaces in other areas such as quotas, please open a separate issue so we can troubleshoot and track efficiently. |
This issue is a bit all over the place. Please provide:
Based on what we have observed so far, there are certainly pvc mount failures in the event log for a test account or two -- but these are related to a race condition between performing a tenant reset and attempting to spin up new resources. Tenant reset and adding objects are most certainly not synchronous steps. Having the above data will help clarify what the actual issues are here. It is not clear that this should be prioritized as a SEV1 given the current lack of information -- rather, this looks more like a SEV3 as an entire business unit is not halted. |
Issue Overview
For our both user accounts on 1a cluster, Jenkins is stuck in idled state and even the reset of environment does not help.
Expected Behaviour
The Jenkins is unidled when pipeline starts.
Current Behaviour
Jenkins is not unidled when pipeline starts. Reset of environment does not help.
Additional Information
This happens only for our 1a accounts. I do not have any personal account on 1a cluster so I cannot verify if this affects all accounts.
The text was updated successfully, but these errors were encountered: