Add HEKETI_IGNORE_STALE_OPERATIONS env variable in heketi. Helps to d… #447

SaravanaStorageNetwork · 2018-03-05T10:10:58Z

…ecide whether to ignore the stale pending operations db entries in the database.

Default value is true.

Signed-off-by: Saravanakumar sarumuga@redhat.com

This change is

centos-ci · 2018-03-05T10:15:02Z

Can one of the admins verify this patch?

obnoxxx · 2018-03-05T10:21:55Z

ok to test

obnoxxx

Just a question about the doc text in the ocp templates (would apply to both heketi and deploy-heketi templates...)

obnoxxx · 2018-03-05T10:23:32Z

deploy/ocp-templates/heketi-template.yaml

@@ -121,3 +123,6 @@ parameters:
  displayName: heketi fstab path
  description: Set the fstab path, file that is populated with bricks that heketi creates
  value: /var/lib/heketi/fstab
+- name: HEKETI_IGNORE_STALE_OPERATIONS
+  displayName: Heketi ignore stale operations


"Heketi ignore stale operations" --> "Whether to ignore stale operations at startup" ?

obnoxxx · 2018-03-05T10:29:22Z

deploy/ocp-templates/heketi-template.yaml

@@ -121,3 +123,6 @@ parameters:
  displayName: heketi fstab path
  description: Set the fstab path, file that is populated with bricks that heketi creates
  value: /var/lib/heketi/fstab
+- name: HEKETI_IGNORE_STALE_OPERATIONS
+  displayName: Heketi ignore stale operations
+  description: Whether to ignore the stale pending operations db entries in the database


Should we be a little more verbose?

"This allows to control whether heketi should start up when when there are stale pending operation entries present in the database."

Maybe even more (not sure how long this can/should be): " ... Setting this to true lets heketi ignore existing pending operations at startup. Setting it to false causes heketi to refuse to start if pending operations are found in the database."

obnoxxx

As discussed, please set the value to "True" in the templates.

SaravanaStorageNetwork · 2018-03-05T10:42:53Z

@obnoxxx @raghavendra-talur Please check

obnoxxx

The resulting templates LGTM now, thanks for the update!

But I have a nitpick about the commits:

The first adds the var and an initial description.
The second commit changes the value (which would be OK), but also changes (elaborates on) the description...

I suggest squashing the two commit or else reworking it such that the second only changes the values from true to false...

raghavendra-talur · 2018-03-05T11:15:58Z

@SaravanaStorageNetwork Also, looks like we need to have the value as string and bool. Enclose true in quotes.

obnoxxx · 2018-03-05T11:28:42Z

retest this please

obnoxxx

LGTM now.
let's see what centos-ci has to say about this. :-)

obnoxxx · 2018-03-05T11:29:48Z

ok to test

obnoxxx · 2018-03-05T12:08:22Z

@SaravanaStorageNetwork centos-ci failed:

11:54:20 heketi topology loaded.
11:54:20 /usr/bin/kubectl -n default exec -i deploy-heketi-6db75b9566-vrtg4 -- heketi-cli -s http://localhost:8080 --user admin --secret '' setup-openshift-heketi-storage --listfile=/tmp/heketi-storage.json  2>&1
11:55:29 Saving /tmp/heketi-storage.json
11:55:29 /usr/bin/kubectl -n default exec -i deploy-heketi-6db75b9566-vrtg4 -- cat /tmp/heketi-storage.json | /usr/bin/kubectl -n default create -f - 2>&1
12:01:27 Error from server: timeout
12:01:27 Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (post endpoints)
12:01:27 Unable to connect to the server: http2: server sent GOAWAY and closed the connection; LastStreamID=59, ErrCode=NO_ERROR, debug=""; some request body already written
12:01:27 The connection to the server 192.168.10.90:6443 was refused - did you specify the right host or port?
12:01:27 Failed on creating heketi storage resources.

…ecide whether to ignore the stale pending operations db entries in the database. Default value is True. Signed-off-by: Saravanakumar <sarumuga@redhat.com>

SaravanaStorageNetwork · 2018-03-05T12:49:16Z

ok to test

SaravanaStorageNetwork · 2018-03-05T13:30:35Z

13:15:33 Error from server (Timeout): error when creating "myapp.yaml": the server was unable to return a response in the time allotted, but may still be processing the request (post pods)
13:15:33 waiting for app pod to become available
13:15:45 putting content into application
13:15:54 Error from server (NotFound): pods "myapp" not found
13:15:54 verifying we get back our content from the app
13:15:54 % Total % Received % Xferd Average Speed Time Time Time Current
13:15:54 Dload Upload Total Spent Left Speed
13:15:54
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: http; Name or service not known
13:15:54 ERROR: did not get expected output from nginx pod

nginx pod failing ? seems like network issue.. @obnoxxx

SaravanaStorageNetwork · 2018-03-05T13:41:59Z

ok to test

obnoxxx · 2018-03-05T13:46:55Z

retest this please

obnoxxx · 2018-03-05T13:47:40Z

ok to test

obnoxxx · 2018-03-05T13:49:32Z

why is the @centos-ci not reacting to the magic incantations? ... :-/

obnoxxx · 2018-03-05T13:50:20Z

hah, it just takes a short while! :-)

SaravanaStorageNetwork · 2018-03-05T14:20:30Z

There is no change between previous run and this one - but failed during deploy-heketi - Could you retrigger @obnoxxx

jarrpa · 2018-03-05T14:39:41Z

retest this please

obnoxxx · 2018-03-05T15:39:25Z

This is weird. Seems to be failing arbitrarily at different stages of the test run. Is it related at all?

raghavendra-talur · 2018-03-05T15:41:49Z

@obnoxxx i don't think. So network error or a package is broken.

SaravanaStorageNetwork · 2018-03-05T16:09:45Z

@obnoxxx I have verified the test steps works in my local setup - but fails in CI.

I have sent another PR to verify whether any issue with env variable value true. #448

If that also fails, some issue with CI.
else, issue with using "true" value which we added here.

obnoxxx · 2018-03-05T16:26:00Z

Also created PR #449 to test the centos-ci. That PR just changes a readme...

jarrpa · 2018-03-05T16:45:02Z

retest this please

obnoxxx · 2018-03-05T16:48:38Z

hey, did it actually already pick up a heketi build with the HEKETI_IGNORE_STALE_OPERATIONS variable support? with value true/false?

raghavendra-talur · 2018-03-05T16:52:13Z

hey, did it actually already pick up a heketi build with the HEKETI_IGNORE_STALE_OPERATIONS variable support? with value true/false?

Whether it did or not should not matter. Isn't it?

obnoxxx · 2018-03-05T16:53:17Z

ugh #448 succeeded...

obnoxxx · 2018-03-05T17:17:39Z

hooray, it passed!

obnoxxx suggested changes Mar 5, 2018

View reviewed changes

raghavendra-talur approved these changes Mar 5, 2018

View reviewed changes

obnoxxx suggested changes Mar 5, 2018

View reviewed changes

SaravanaStorageNetwork force-pushed the add_ignore_stale_entry branch from 20dfe26 to cede6f8 Compare March 5, 2018 11:25

obnoxxx approved these changes Mar 5, 2018

View reviewed changes

Add HEKETI_IGNORE_STALE_OPERATIONS env variable in heketi. Helps to d…

867224b

…ecide whether to ignore the stale pending operations db entries in the database. Default value is True. Signed-off-by: Saravanakumar <sarumuga@redhat.com>

SaravanaStorageNetwork force-pushed the add_ignore_stale_entry branch from cede6f8 to 867224b Compare March 5, 2018 12:48

SaravanaStorageNetwork mentioned this pull request Mar 5, 2018

TEST PR - DO NOT MERGE #448

Closed

obnoxxx merged commit 0b27a92 into gluster:master Mar 5, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HEKETI_IGNORE_STALE_OPERATIONS env variable in heketi. Helps to d… #447

Add HEKETI_IGNORE_STALE_OPERATIONS env variable in heketi. Helps to d… #447

SaravanaStorageNetwork commented Mar 5, 2018 •

edited

centos-ci commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx left a comment

obnoxxx Mar 5, 2018

obnoxxx Mar 5, 2018

obnoxxx left a comment

SaravanaStorageNetwork commented Mar 5, 2018

obnoxxx left a comment

raghavendra-talur commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx left a comment

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

jarrpa commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

raghavendra-talur commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

jarrpa commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

raghavendra-talur commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

Add HEKETI_IGNORE_STALE_OPERATIONS env variable in heketi. Helps to d… #447

Add HEKETI_IGNORE_STALE_OPERATIONS env variable in heketi. Helps to d… #447

Conversation

SaravanaStorageNetwork commented Mar 5, 2018 • edited

centos-ci commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx left a comment

Choose a reason for hiding this comment

obnoxxx Mar 5, 2018

Choose a reason for hiding this comment

obnoxxx Mar 5, 2018

Choose a reason for hiding this comment

obnoxxx left a comment

Choose a reason for hiding this comment

SaravanaStorageNetwork commented Mar 5, 2018

obnoxxx left a comment

Choose a reason for hiding this comment

raghavendra-talur commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx left a comment

Choose a reason for hiding this comment

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

jarrpa commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

raghavendra-talur commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

jarrpa commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

raghavendra-talur commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

obnoxxx commented Mar 5, 2018

SaravanaStorageNetwork commented Mar 5, 2018 •

edited