Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HPCC-17581 Persistent connection support for ESP #10971

Merged
merged 2 commits into from May 3, 2018

Conversation

Projects
None yet
5 participants
@mayx
Copy link
Member

commented Mar 15, 2018

  • Add IPersistentHandler interface and class to keep and maintain
    persistent connections
  • Add persistent connection to server side ESP. Add maxPersistentIdleTime
    and maxPersistentRequest configuration parameters.
  • Add persistent connection to httpclient that is used by ws_ecl and
    esdl to connect to Roxie and other services.
  • Add -persist option to soapplus client and server commands

Signed-off-by: mayx yanrui.ma@lexisnexisrisk.com

Type of change:

  • This change is a bug fix (non-breaking change which fixes an issue).
  • This change is a new feature (non-breaking change which adds functionality).
  • This change improves the code (refactor or other change that does not change the functionality)
  • This change fixes warnings (the fix does not alter the functionality or the generated code)
  • This change is a breaking change (fix or feature that will cause existing behavior to change).
  • This change alters the query API (existing queries will have to be recompiled)

Checklist:

  • My code follows the code style of this project.
    • My code does not create any new warnings from compiler, build system, or lint.
  • The commit message is properly formatted and free of typos.
    • The commit message title makes sense in a changelog, by itself.
    • The commit is signed.
  • My change requires a change to the documentation.
    • I have updated the documentation accordingly, or...
    • I have created a JIRA ticket to update the documentation.
    • Any new interfaces or exported functions are appropriately commented.
  • I have read the CONTRIBUTORS document.
  • The change has been fully tested:
    • I have added tests to cover my changes.
    • All new and existing tests passed.
    • I have checked that this change does not introduce memory leaks.
    • I have used Valgrind or similar tools to check for potential issues.
  • I have given due consideration to all of the following potential concerns:
    • Scalability
    • Performance
    • Security
    • Thread-safety
    • Premature optimization
    • Existing deployed queries will not be broken
    • This change fixes the problem, not just the symptom
    • The target branch of this pull request is appropriate for such a change.
  • There are no similar instances of the same problem that should be addressed
    • I have addressed them here
    • I have raised JIRA issues to address them separately
  • This is a user interface / front-end modification
    • I have tested my changes in multiple modern browsers
    • The component(s) render as expected

Testing:

@hpcc-jirabot

This comment has been minimized.

Copy link

commented Mar 15, 2018

@mayx

This comment has been minimized.

Copy link
Member Author

commented Mar 15, 2018

@afishbeck please review.

@mayx mayx force-pushed the mayx:HPCC-17581-PersistentConn branch from cb8b309 to 575ed64 Apr 12, 2018

@afishbeck
Copy link
Member

left a comment

@mayx some fairly minor comments.

m_socket->shutdown();
m_socket->close();
m_socket->Release();
m_socket = NULL;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

might as well switch to nullptr, since you've touched it.

ERRLOG("Error connecting to %s", ep.getUrlStr(url).str());
DBGLOG(e);
e->Release();
m_socket = NULL;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

nullptr

{
StringBuffer url;
ERRLOG("Unknown exception connecting to %s", ep.getUrlStr(url).str());
m_socket = NULL;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

nullptr

@@ -100,6 +102,10 @@ private:
StringAttr m_password;
StringAttr m_realm;
ISecureSocketContext *m_ssctx;
IPersistentHandler* m_persistentHandler;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

better to initialize all of these new member variables here, rather than in the constructor.

@@ -389,23 +402,23 @@ const char * CSecureHttpProtocol::getProtocolName()
* CHttpThread Implementation *
**************************************************************************/
CHttpThread::CHttpThread(bool viewConfig) :
CEspProtocolThread("Http Thread")
CEspProtocolThread("Http Thread"), m_persistentHandler(nullptr)

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

For new variables better to init in class declaration rather than constructor.

float httpver = 1.0;
if(m_version.length() > 5)
httpver = atof(m_version.str()+5);
if(((httpver + 0.001 > 1.1 && (conheader.length() == 0 || stricmp(conheader.str(), "Close") != 0))

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

I think this could be simplified / made easier to read. If only "1.0" used the close by default behavior then maybe you could just check for that string rather than using float. also breaking up the statement could be more readable. Maybe an inline function could simplify even more. Short circuiting the logic and returning immediately when the answer is known.

@@ -71,6 +71,8 @@ protected:
StringAttr m_paramstr;
int m_supportClientXslt;
bool m_isForm;
bool m_persistentEligible;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

init default to match old behavior here?

@@ -99,8 +100,9 @@ class CEspApplicationPort

HINSTANCE hxsl;
Owned<IXslProcessor> xslp;
CEspProtocol* protocol;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

init to nullptr here?

@@ -35,7 +35,7 @@ class CEspProtocolThread: public Thread
StringAttr m_name;

int run();

bool keepAlive;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

init to false here?

Owned<ISocketSelectHandler> m_selectHandler;
IPersistentSelectNotify* m_notify;
Semaphore m_waitsem;
bool m_stop;

This comment has been minimized.

Copy link
@afishbeck

afishbeck Apr 30, 2018

Member

Init m_stop and whatever else appropriate here.

@mayx mayx force-pushed the mayx:HPCC-17581-PersistentConn branch from 575ed64 to a45a3c2 May 2, 2018

@mayx

This comment has been minimized.

Copy link
Member Author

commented May 2, 2018

@afishbeck Changes checked in as separate commit, please review...

@mayx mayx force-pushed the mayx:HPCC-17581-PersistentConn branch 4 times, most recently from 6ee78b2 to 72c7ef5 May 2, 2018

HPCC-17581 Adding ESP Server side persistent connection support
- Add IPersistentHandler interface and class to keep and maintain
  persistent connections
- Add persistent connection to server side ESP. Add maxPersistentIdleTime
  and maxPersistentRequest configuration parameters.
- Add persistent connection to httpclient that is used by ws_ecl and
  esdl to connect to Roxie and other services.
- Add -persist option to soapplus client and server commands

Signed-off-by: mayx <yanrui.ma@lexisnexisrisk.com>

@mayx mayx force-pushed the mayx:HPCC-17581-PersistentConn branch from 72c7ef5 to bcdba0b May 3, 2018

HPCC-17581 Persistent connection support for ESP
- Changes based on code review comments
- Set persistent log level according to esp's log level
- Don't reuse connection if response is empty

Signed-off-by: mayx <yanrui.ma@lexisnexisrisk.com>

@mayx mayx force-pushed the mayx:HPCC-17581-PersistentConn branch from bcdba0b to 79d7d66 May 3, 2018

@HPCCSmoketest

This comment has been minimized.

Copy link
Contributor

commented May 3, 2018

Automated Smoketest:
OS: centos 7.2.1511 (Linux 3.10.0-327.28.3.el7.x86_64)
Sha: 79d7d66
Build: success
Install hpccsystems-platform-community_6.5.0-trunk0.el7.x86_64.rpm
HPCC Start: OK

Unit tests result:

Test total passed failed errors timeout
unittest 88 88 0 0 0
wutoolTest(Dali) 19 19 0 0 0
wutoolTest(Cassandra) 19 19 0 0 0

Regression test result:

phase total pass fail
setup (hthor) 11 11 0
setup (thor) 11 11 0
setup (roxie) 11 11 0
test (hthor) 780 780 0
test (thor) 711 711 0
test (roxie) 824 824 0

HPCC Stop: OK
HPCC Uninstall: OK
Time stats:

Prep time Build time Package time Install time Start time Test time Stop time Summary
15 sec (00:00:15) 187 sec (00:03:07) 56 sec (00:00:56) 5 sec (00:00:05) 21 sec (00:00:21) 1159 sec (00:19:19) 17 sec (00:00:17) 1460 sec (00:24:20)
@afishbeck
Copy link
Member

left a comment

@mayx looks good

@richardkchapman richardkchapman merged commit 5956210 into hpcc-systems:master May 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.