Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cluster is ready even if re-import fails #827

Closed
ltrilety opened this issue Feb 20, 2018 · 12 comments
Closed

cluster is ready even if re-import fails #827

ltrilety opened this issue Feb 20, 2018 · 12 comments
Assignees

Comments

@ltrilety
Copy link
Contributor

In some cases it could happen that cluster looks like ready and fine even when import task fails.
Test steps:

  1. import cluster - cluster is imported successfully without issue
  2. un-manage cluster - cluster is successfully un-managed
  3. on some gluster server node remove tendrl-gluster-integration package and break the tendrl repository file
  4. run import again

import task failed however cluster looks like imported and ready

import task details
import-failed

cluster list
cluster_list

Tested version:
tendrl-commons-1.6.0-1.el7.centos.noarch
tendrl-api-1.6.0-1.el7.centos.noarch
tendrl-ui-1.6.0-1.el7.centos.noarch
tendrl-grafana-selinux-1.5.4-2.el7.centos.noarch
tendrl-ansible-1.5.5-1.el7.centos.noarch
tendrl-notifier-1.6.0-1.el7.centos.noarch
tendrl-node-agent-1.6.0-1.el7.centos.noarch
tendrl-api-httpd-1.6.0-1.el7.centos.noarch
tendrl-selinux-1.5.4-2.el7.centos.noarch
tendrl-grafana-plugins-1.6.0-1.el7.centos.noarch
tendrl-monitoring-integration-1.6.0-1.el7.centos.noarch

@gnehapk
Copy link
Member

gnehapk commented Feb 20, 2018

@ltrilety Can you please share the API response of \clusters.

@ltrilety
Copy link
Contributor Author

ltrilety commented Feb 20, 2018

@gnehapk I already made some changes and when I tried the scenario again I don't get the same state. This time I have unhealthy cluster state. All other is the same. The cluster is still Ready to Use.
Anyway clusters API response:

clusters | […]
-- | --
0 | {…}
1 | {…}
  updated_at | 2018-02-20 13:00:05.470531+00:00
  sds_name | gluster
  cluster_name | gluster-735ad20308a447813aa4cc2ea540775fde1759afb69470e0f6fb56976c10b320
  cluster_id | d004158e-4b08-426d-90ec-0f20aeb5e892
  integration_id | d004158e-4b08-426d-90ec-0f20aeb5e892
  hash | 7242cea496f08376649edb449f568d92
  sds_version | 4.1dev
  errors | […]
    0 | Atom  Execution failed. Error: Error executing atom:  tendrl.objects.Cluster.atoms.ImportCluster on 
  flow: Import existing  Gluster Cluster
  globaldetails | {…}
    status | unhealthy
  public_network |  
  cluster_network |  
  is_managed | yes
  enable_volume_profiling | yes
  alert_counters | {…}
    hash | c1610b01cbfebe40a48f6442b14b02dc
    warning_count | 2
    updated_at | 2018-02-20 13:01:55.388690+00:00
    integration_id | d004158e-4b08-426d-90ec-0f20aeb5e892
  status |  
  current_job | {"status": "finished", "job_id": "b1595ff3-ac11-4bb5-83be-908e303a3587", "job_name": "ImportCluster"}
  nodes | […]
    0 | {…}
      updated_at | 2018-02-20 13:21:40.942683+00:00
      node_id | 322de34c-5dfe-4352-89c8-3df105e9511e
      last_sync | 2018-02-20 13:20:40.662539+00:00
      status | UP
      hash | 3e6d0214e160cba309b16c4fa4006e70
      tags | […]
      sync_status | in_progress
      fqdn | ltrilety-usm1-gl1.usmqe.lab.eng.brq.redhat.com
    1 | {…}
...
    2 | {…}
...
    3 | {…}
...
    4 | {…}
...
    5 | {…}
...

@gnehapk
Copy link
Member

gnehapk commented Feb 26, 2018

@ltrilety Can you please close this issue if you are not seeing this issue any more.

@ltrilety
Copy link
Contributor Author

@gnehapk I can close it if it will not be valid in another release. However in this release it's valid. What I said in my previous comment was not that I don't see the issue. I just said that when I repeated reproduction steps the cluster state was different - not healthy but unhealthy. That doesn't have any impact on issue validity.

@gnehapk
Copy link
Member

gnehapk commented Feb 26, 2018

@ltrilety Ack. @r0h4n @nthomas-redhat @shtripat @GowthamShanmugam is_managed is set to "yes" in the response shared by @ltrilety, hence UI is displaying it is managed cluster. Please check.

@gnehapk
Copy link
Member

gnehapk commented Mar 8, 2018

@r0h4n @shtripat @nthomas-redhat Please close the issue, if its fixed.

@r0h4n r0h4n removed this from the Milestone 3 (2018) milestone Mar 8, 2018
@r0h4n
Copy link
Contributor

r0h4n commented Mar 8, 2018

@shtripat We need to set GlobalDetails.status as unhealthy and Cluster.status appropriately in case of import job failures.

@r0h4n r0h4n closed this as completed Mar 8, 2018
@r0h4n r0h4n added this to the Milestone 3 (2018) milestone Mar 8, 2018
@ltrilety
Copy link
Contributor Author

ltrilety commented Mar 8, 2018

I tried to import cluster with not functional repository set so it has to fail. The fail happened however the cluster looks like ready and healthy.

Tested version:
tendrl-ui-1.6.1-1.el7.centos.noarch

Reproduction steps:

  1. Prepare gluster cluster, prepare tendrl server and run tendrl-ansible
  2. Broke the tendrl.repo file on some gluster node
    e.g. change baseurl line to baseurl = http://bad.org/results/tendrl/release/epel-7-x86_64/
  3. Try to import the cluster from tendrl UI

Actual result:
The import failed, however cluster seems to be managed.

image

BTW: I have another issue, but that could be related with Brno network issues.

@julienlim
Copy link
Member

If import fails, cluster status should show ? (unknown).

@gnehapk
Copy link
Member

gnehapk commented May 30, 2018

@r0h4n @shtripat @nthomas-redhat Can you please close this issue, if fixed.

@gnehapk gnehapk removed this from the Milestone 6 (2018) milestone May 30, 2018
@gnehapk
Copy link
Member

gnehapk commented Sep 11, 2018

@GowthamShanmugam Please provide update for this issue.

@GowthamShanmugam
Copy link
Contributor

GowthamShanmugam commented Sep 11, 2018

This issue is fixed, we can close this issue. @ltrilety verify with the latest master and reopen this issue if it is reproducible.

@gnehapk gnehapk closed this as completed Sep 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants