Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support https for es client #3150

Merged

Conversation

bright-starry-sky
Copy link
Contributor

Support https for elasticsearch client, such as :
curl -u elastic:123456 -k https://127.0.0.1:9200/_cat/indices?v

syntax:

SIGN IN TEXT SERVICE (127.0.0.1:9200, HTTP);
SIGN IN TEXT SERVICE (127.0.0.1:9200, HTTP, "user", "password");
SIGN IN TEXT SERVICE (127.0.0.1:9200, HTTPS, "user", "password");

requirement from :
https://discuss.nebula-graph.com.cn/t/topic/4787

@bright-starry-sky bright-starry-sky added the ready-for-testing PR: ready for the CI test label Oct 19, 2021
@bright-starry-sky bright-starry-sky added this to the v2.6.0 milestone Oct 19, 2021
@Sophie-Xie Sophie-Xie added the cherry-pick-v2.6 PR: need cherry-pick to this version label Oct 19, 2021
@codecov-commenter
Copy link

Codecov Report

Merging #3150 (a4daec0) into master (fb3be74) will increase coverage by 0.71%.
The diff coverage is 82.55%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3150      +/-   ##
==========================================
+ Coverage   84.30%   85.01%   +0.71%     
==========================================
  Files        1287     1289       +2     
  Lines      115636   115920     +284     
==========================================
+ Hits        97483    98549    +1066     
+ Misses      18153    17371     -782     
Impacted Files Coverage Δ
src/clients/storage/InternalStorageClient.cpp 0.00% <0.00%> (ø)
src/graph/executor/admin/ShowTSClientsExecutor.cpp 0.00% <0.00%> (ø)
src/graph/util/FTIndexUtils.cpp 4.12% <0.00%> (-0.05%) ⬇️
src/graph/validator/FetchVerticesValidator.h 100.00% <ø> (ø)
src/graph/validator/Validator.h 96.29% <ø> (ø)
src/kvstore/NebulaStore.h 94.73% <ø> (-0.51%) ⬇️
src/kvstore/Part.h 100.00% <ø> (ø)
src/kvstore/plugins/elasticsearch/ESListener.cpp 0.00% <0.00%> (ø)
.../storage/transaction/ChainAddEdgesProcessorLocal.h 40.00% <ø> (ø)
...orage/transaction/ChainAddEdgesProcessorRemote.cpp 0.00% <0.00%> (ø)
... and 138 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7cd8e67...a4daec0. Read the comment docs.

{
std::string query =
"SIGN IN TEXT SERVICE (127.0.0.1:9200, HTTP, \"user\", \"password\"), "
"(127.0.0.1:9200, HTTPS, \"user\", \"password\")";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use following format:

SIGN IN TEXT SERVICE 
  (http://127.0.0.1:9200, "user", "password"),
  (https://127.0.0.1:9200, "user", "password")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not use following format:

SIGN IN TEXT SERVICE 
  (http://127.0.0.1:9200, "user", "password"),
  (https://127.0.0.1:9200, "user", "password")

Good idea, I agree with you. If we change it now, there will be compatibility issues. I suggest improvements in the next big release, WDYT ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why exist incompatible issue? We can use http by default if user don't specify it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why exist incompatible issue? We can use http by default if user don't specify it.

I want to change the meta data storage by thrift structure , current as below :

struct FTClient {
    1: required common.HostAddr    host,
    2: optional binary             user,
    3: optional binary             pwd,
    4: optional binary             conn_type,
}

I think below is better :

struct FTClient {
    1: required binary             conn_prefix,  <----- "http://127.0.0.1:9200"
    2: optional binary             user,
    3: optional binary             pwd,
}

This change needs to be handled by the upgrade tool.

Copy link
Contributor

@panda-sheep panda-sheep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!


explicit HttpClient(const HostAddr& h) noexcept : host(h) {}
explicit HttpClient(const HostAddr& h) noexcept : host(h) { connType = "http"; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line 42, directly set the default value of connType to "http", wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line 42, directly set the default value of connType to "http", wdyt?

Good point.

@@ -98,7 +98,7 @@ TEST(FulltextPluginTest, ESBulkTest) {
auto header = ESStorageAdapter().bulkHeader(hc);
std::string expected =
"/usr/bin/curl -H \"Content-Type: application/x-ndjson; "
"charset=utf-8\" -XPOST \"http://127.0.0.1:9200/_bulk\"";
"charset=utf-8\" -XPOST -k \"http://127.0.0.1:9200/_bulk\"";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-k allow curl to use non-secure ssl connection and transfer data ?

HttpClient(const HostAddr& h,
const std::string& u,
const std::string& p,
const std::string&& c) noexcept
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const std::string&& doesn't seem to work std::string&& is ok ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const std::string&& doesn't seem to work std::string&& is ok ?

Good point, negligent.

@bright-starry-sky
Copy link
Contributor Author

addressed comments

@bright-starry-sky
Copy link
Contributor Author

close this PR, further discussion is needed.

@yixinglu yixinglu merged commit 4886d4f into vesoft-inc:master Nov 23, 2021
@Sophie-Xie Sophie-Xie removed the cherry-pick-v2.6 PR: need cherry-pick to this version label Nov 23, 2021
@Sophie-Xie Sophie-Xie added the doc affected PR: improvements or additions to documentation label Jan 4, 2022
@Sophie-Xie Sophie-Xie removed this from the v2.6.0 milestone Jan 17, 2022
@Sophie-Xie Sophie-Xie added this to To do in Nebula Docs Jan 17, 2022
@cooper-lzy cooper-lzy moved this from To do to Done in Nebula Docs Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc affected PR: improvements or additions to documentation ready-for-testing PR: ready for the CI test
Projects
Development

Successfully merging this pull request may close these issues.

None yet

7 participants