HPCC-20493 Allow connect and read timeout to be configured in ESP #12392

shamser · 2019-04-04T10:24:40Z

Signed-off-by: Shamser Ahmed shamser.ahmed@lexisnexis.co.uk

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Testing:

hpcc-jirabot · 2019-04-04T10:24:49Z

https://track.hpccsystems.com/browse/HPCC-20493
Jira updated

shamser · 2019-04-04T10:25:44Z

TODO: still working on providing the test cases for this.

shamser · 2019-04-08T14:24:18Z

@afishbeck Please can you review these changes relating being able to set the timeouts for ESP services. By the way, I've tested this manually but (1) created a multi-node thor (2) spraying to this thor (3) shutting down the remote thor node's dafilesrv (4) executing WUInfo, DFUQuery and DFUInfo ESP queries. Compare how long it takes to timeout with and without the new settings. (I'm inclined not produce test cases because it seems to be quite a lot of work for not too much benefit)

afishbeck · 2019-04-09T19:34:21Z

esp/platform/espcfg.cpp

@@ -325,6 +326,10 @@ CEspConfig::CEspConfig(IProperties* inputs, IPropertyTree* envpt, IPropertyTree*
        if (m_cfg->getProp("@daliServers", daliservers))
            initDali(daliservers.str()); //won't init if detached

+        const unsigned dafilesrvConnectTimeout = m_cfg->getPropInt("@dafilesrvConnectTimeout", 2000);


How were the defaults decided on? Will a change in default behavior catch the users off guard? Looks like old behavior was the equivalent of 0's here?

We discussed the default timeout period last Monday. @jakesmith suggested that having shorter timeout period would mean be sensible (resolves issues such as https://track.hpccsystems.com/browse/HPCC-19002) and if it were to cause problems, the timeout period could be extended with the configuration option.

I agree that it would cause problems if say a 30 second timeout is necessary and eventually returns a results. But does any remote file access require this length of time? May be 2 seconds timeout is too short?

@jakesmith ?

My suggestion was that the global default (not just in esp) should be much shorter than it is now.
( Currently the defaults are 100 seconds and 3 retries. )
And it should be globally configurable (e.g. via environment.conf), so that if needed / if problematic, it can be raised.

I think I had suggested changing it to 10 seconds (3 retries may be still ok).

Once that is done, whilst having a separate configuration in Esp is okay, it becomes less relevant.

There is also the further improvement of implementing a better mechanism via a timed blacklist, so clients don't keep retrying the same endponit and repeatedly timing out, which cumulatively can be significant even if short. (There is an implementation like this in thorsoapcall, which should be looked at to see whether it can be reused).
@shamser - was a JIRA opened for the blacklist implementation ?

@jakesmith I have just created the blackout jira: https://track.hpccsystems.com/browse/HPCC-21927. @afishbeck Do you think it should be set to '0' or is 10 seconds ok?

I have just created the blackout jira

@shamser - blacklist?

Do you think it should be set to '0' or is 10 seconds ok?

@shamser - are you asking re. this esp custom timeout, or the global timeout I was talking about above?
Has a JIRA been opened to change the global timeout?

If the global timeout is set as suggested (10 secs) - then I'd default the esp timeout to unset , will default to global timeout - but they (ops) can override it if they want to.

@jakesmith I'm talking about the ESP custom and default timeouts. This one is designed to address a number of issues reported relating to ESP services (making remote file requests) taking too long to timeout. As far as other aspects related to timeouts, I feel that i may not be immediately pressing as no has complained (yet) and we can merge this one until we have agreed the more comprehensive approach for timings. By the way, there was a bug in the previous commit, which I have now fixed and pushed. (I've also set the default timeout period to 10 seconds).

afishbeck

@shamser one question.

jakesmith · 2019-04-15T08:52:58Z

initfiles/componentfiles/configschema/xsd/esp.xsd

-
+                <xs:attribute name="dafilesrvConnectTimeout" type="xs:nonNegativeInteger"
+                              hpcc:displayName="Remote file connection timeout in seconds"
+                              hpcc:presentValue="2"


this should be 10 now ?

jakesmith · 2019-04-15T08:53:02Z

initfiles/componentfiles/configschema/xsd/esp.xsd

+                              hpcc:tooltip="Remote file access connection timeout in seconds"/>
+                <xs:attribute name="dafilesrvReadTimeout" type="xs:nonNegativeInteger"
+                              hpcc:displayName="Remote file read timeout in seconds"
+                              hpcc:presentValue="2"


this should be 10 now ?

jakesmith · 2019-04-15T08:53:51Z

@shamser - 1 comment. Please go ahead and squash after changing.

Signed-off-by: Shamser Ahmed <shamser.ahmed@lexisnexis.co.uk>

shamser · 2019-04-17T11:45:52Z

@jakesmith Squashed. Please can you do final review.

HPCCSmoketest · 2019-04-17T12:08:51Z

Automated Smoketest: ✅
OS: centos 7.4.1708 (Linux 3.10.0-327.28.3.el7.x86_64)
Sha: ea8c114
Build: success
Build: success
Install HPCC Platform
HPCC Start: OK

Unit tests result:

Test	total	passed	elaps
unittest	111	111	35 sec
wutoolTest(Dali)	19	19	2 sec
wutoolTest(Cassandra)	19	19	7 sec

Regression test result:

phase	total	pass	elaps
setup (hthor)	11	11	23 sec (00:00:23)
setup (thor)	11	11	44 sec (00:00:44)
setup (roxie)	11	11	27 sec (00:00:27)
test (hthor)	827	827	196 sec (00:03:16)
test (thor)	752	752	608 sec (00:10:08)
test (roxie)	903	903	223 sec (00:03:43)

HPCC Stop: OK
Time stats:

Prep time	Build time	Package time	Install time	Start time	Test time	Stop time	Summary
29 sec (00:00:29)	209 sec (00:03:29)	0 sec (00:00:00)	3 sec (00:00:03)	19 sec (00:00:19)	1319 sec (00:21:59)	21 sec (00:00:21)	1600 sec (00:26:40)

jakesmith · 2019-04-17T13:39:21Z

Looks good.

@richardkchapman - ready to merge.

richardkchapman · 2019-04-18T07:38:38Z

@JamesDeFabia @shamser Is there a documentation impact?

shamser · 2019-04-18T10:10:10Z

@JamesDeFabia @shamser Is there a documentation impact?

@JamesDeFabia @richardkchapman I have created a documentation Jira: https://track.hpccsystems.com/browse/HPCC-21974. This change affects current behaviour. How do we get this documented in the Redbook?

richardkchapman · 2019-04-18T10:18:41Z

I have added a tag

shamser force-pushed the issue20493 branch 3 times, most recently from 3674095 to 49b24b5 Compare April 4, 2019 15:34

afishbeck reviewed Apr 9, 2019

View reviewed changes

jakesmith reviewed Apr 15, 2019

View reviewed changes

shamser force-pushed the issue20493 branch from 1737ba5 to e2969d6 Compare April 17, 2019 11:38

HPCC-20493 Allow connect and read timeout to be configured in ESP

ea8c114

Signed-off-by: Shamser Ahmed <shamser.ahmed@lexisnexis.co.uk>

shamser force-pushed the issue20493 branch from e2969d6 to ea8c114 Compare April 17, 2019 11:39

richardkchapman merged commit e656939 into hpcc-systems:master Apr 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPCC-20493 Allow connect and read timeout to be configured in ESP #12392

HPCC-20493 Allow connect and read timeout to be configured in ESP #12392

shamser commented Apr 4, 2019 •

edited

Loading

hpcc-jirabot commented Apr 4, 2019

shamser commented Apr 4, 2019

shamser commented Apr 8, 2019

afishbeck Apr 9, 2019

shamser Apr 11, 2019

jakesmith Apr 11, 2019 •

edited

Loading

shamser Apr 11, 2019

jakesmith Apr 11, 2019 •

edited

Loading

jakesmith Apr 11, 2019

shamser Apr 12, 2019

afishbeck left a comment

jakesmith Apr 15, 2019

jakesmith Apr 15, 2019

jakesmith commented Apr 15, 2019

shamser commented Apr 17, 2019

HPCCSmoketest commented Apr 17, 2019

jakesmith commented Apr 17, 2019

richardkchapman commented Apr 18, 2019

shamser commented Apr 18, 2019

richardkchapman commented Apr 18, 2019

HPCC-20493 Allow connect and read timeout to be configured in ESP #12392

HPCC-20493 Allow connect and read timeout to be configured in ESP #12392

Conversation

shamser commented Apr 4, 2019 • edited Loading

Type of change:

Checklist:

Testing:

hpcc-jirabot commented Apr 4, 2019

shamser commented Apr 4, 2019

shamser commented Apr 8, 2019

afishbeck Apr 9, 2019

Choose a reason for hiding this comment

shamser Apr 11, 2019

Choose a reason for hiding this comment

jakesmith Apr 11, 2019 • edited Loading

Choose a reason for hiding this comment

shamser Apr 11, 2019

Choose a reason for hiding this comment

jakesmith Apr 11, 2019 • edited Loading

Choose a reason for hiding this comment

jakesmith Apr 11, 2019

Choose a reason for hiding this comment

shamser Apr 12, 2019

Choose a reason for hiding this comment

afishbeck left a comment

Choose a reason for hiding this comment

jakesmith Apr 15, 2019

Choose a reason for hiding this comment

jakesmith Apr 15, 2019

Choose a reason for hiding this comment

jakesmith commented Apr 15, 2019

shamser commented Apr 17, 2019

HPCCSmoketest commented Apr 17, 2019

jakesmith commented Apr 17, 2019

richardkchapman commented Apr 18, 2019

shamser commented Apr 18, 2019

richardkchapman commented Apr 18, 2019

shamser commented Apr 4, 2019 •

edited

Loading

jakesmith Apr 11, 2019 •

edited

Loading

jakesmith Apr 11, 2019 •

edited

Loading