curvefs/client: make user agent configurable #2501

Ziy1-Tan · 2023-05-25T07:16:40Z

What problem does this PR solve?

Issue Number: #2499

Problem Summary:

Let user agent item configurable for users having security reasons

What is changed and how it works?

What's Changed:

How it Works:

Side effects(Breaking backward compatibility? Performance regression?):

Check List

Relevant documentation/comments is changed or added
I acknowledge that all my contributions will be made under the project's license

Ziy1-Tan · 2023-05-25T07:18:44Z

cicheck

Cyber-SiKu

Have you tested that it works?

Cyber-SiKu · 2023-05-25T07:18:25Z

curve-ansible/roles/generate_config/templates/s3.conf.j2

@@ -10,7 +10,7 @@ s3.sk={{ s3_sk }}
 # http = 0, https = 1
 s3.http_scheme={{ s3_http_scheme }}
 s3.verify_SSL={{ s3_verify_ssl }}
-s3.user_agent_conf={{ s3_user_agent_conf }}


Ansible is no longer maintained and can not be modified

Cyber-SiKu · 2023-05-25T07:18:32Z

curve-ansible/roles/generate_config/defaults/main.yml

@@ -265,7 +265,7 @@ nebd_server_response_return_rpc_when_io_error: false
 # s3配置默认值
 s3_http_scheme: 0
 s3_verify_ssl: false
-s3_user_agent_conf: S3 Browser


Ziy1-Tan · 2023-05-25T07:39:30Z

cicheck

Ziy1-Tan · 2023-05-25T07:45:00Z

Have you tested that it works?

Yes. I made sure that

s3.user_agent appears in any other configuration item such as s3.verify_SSL.
By debugging TEST_F(S3CompactTest, test_S3AdapterManager), I observed that the userAgent item has been assigned correctly.

Ziy1-Tan · 2023-05-25T07:52:18Z

cicheck

Cyber-SiKu · 2023-05-25T09:03:41Z

Have you tested that it works?

Yes. I made sure that

s3.user_agent appears in any other configuration item such as s3.verify_SSL.

By debugging TEST_F(S3CompactTest, test_S3AdapterManager), I observed that the userAgent item has been assigned correctly.

Please add the corresponding unit test

Ziy1-Tan · 2023-05-26T02:47:52Z

cicheck

Ziy1-Tan · 2023-05-26T03:45:49Z

cicheck

Ziy1-Tan · 2023-05-26T06:05:17Z

cicheck

Ziy1-Tan · 2023-05-29T07:26:07Z

cicheck

wuhongsong · 2023-05-30T03:45:07Z

cicheck

Cyber-SiKu · 2023-05-30T05:50:11Z

curvefs/conf/client.conf
This file should be the configuration file of the client of curvefs and should also need to be modified.

Ziy1-Tan · 2023-05-30T08:21:52Z

cicheck

Ziy1-Tan · 2023-05-30T08:39:43Z

curvefs/conf/client.conf This file should be the configuration file of the client of curvefs and should also need to be modified.

Get it. I'm trying to figure out why the mds election is timeout but I'm failed.

1

wuhongsong · 2023-06-19T06:32:10Z

curvefs/conf/client.conf This file should be the configuration file of the client of curvefs and should also need to be modified.

Get it. I'm trying to figure out why the mds election is timeout but I'm failed.

@SeanHai help

SeanHai · 2023-06-21T01:37:30Z

cicheck

SeanHai · 2023-06-21T02:12:34Z

@Ziy1-Tan

2023-06-21 10:06:04.670070 I | [587688dba922b1f6/10.182.2.26:16701]stop watch current leader
F 2023-06-21T10:06:04.680009+0800 6 s3_adapter.cpp:85]
*** Check failure stack trace: ***
*** Aborted at 1687313164 (unix time) try "date -d @1687313164" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x6) received by PID 6 (TID 0x7fcbd210d4c0) from PID 6; stack trace: ***
@ 0x7fcbd014e0e0 (unknown)
@ 0x7fcbce805fff gsignal
@ 0x7fcbce80742a abort
@ 0x55bec06199ec google::FlushAndAbort()
@ 0x55bec0616c3e google::LogMessage::Fail()
@ 0x55bec0616b87 google::LogMessage::SendToLog()
@ 0x55bec06163b0 google::LogMessage::Flush()
@ 0x55bec0619380 google::LogMessageFatal::~LogMessageFatal()
@ 0x55bebfde1858 curve::common::InitS3AdaptorOptionExceptS3InfoOption()
@ 0x55bebf94fe43 curvefs::mds::MDS::InitFsManagerOptions()
@ 0x55bebf950179 curvefs::mds::MDS::Init()
@ 0x55bebf94c42a main
@ 0x7fcbce7f32e1 __libc_start_main
@ 0x55bebf94bc4a _start
@ 0x0 (unknown)
fileList: []
WARNING: Logging before InitGoogleLogging() is written to STDERR

The curvefs/conf/mds.conf also need add this conf item. If not, it will crash when init s3 adaptor in mds of curvefs.

Ziy1-Tan · 2023-06-21T03:40:56Z

cicheck

Ziy1-Tan · 2023-06-21T04:10:36Z

cicheck

Ziy1-Tan · 2023-06-21T05:13:30Z

cicheck

Ziy1-Tan · 2023-06-21T05:57:52Z

cicheck

Ziy1-Tan · 2023-06-22T03:26:24Z

cicheck

Ziy1-Tan · 2023-06-22T11:10:36Z

cicheck

Ziy1-Tan · 2023-06-22T12:44:13Z

cicheck

Ziy1-Tan · 2023-06-24T06:43:55Z

cicheck

Cyber-SiKu · 2023-06-25T02:51:22Z

cichek

caoxianfei1 · 2023-06-28T06:08:06Z

add more test case @Ziy1-Tan

wuhongsong · 2023-07-05T08:14:30Z

cicheck

wuhongsong · 2023-07-05T08:15:06Z

add more test case @Ziy1-Tan

I have add some test case，lets try it again.

Ziy1-Tan · 2023-07-06T08:57:20Z

cicheck

SeanHai · 2023-07-09T03:27:24Z

cicheck

wuhongsong · 2023-07-17T02:11:00Z

@Ziy1-Tan will do it later, It's a bit busy these days.

Ziy1-Tan · 2023-07-22T07:50:51Z

cicheck

Ziy1-Tan · 2023-07-22T08:03:06Z

cicheck

Signed-off-by: Ziy1-Tan <ajb459684460@gmail.com>

Ziy1-Tan · 2023-07-22T09:14:50Z

cicheck

Ziy1-Tan requested a review from Cyber-SiKu May 25, 2023 07:17

Cyber-SiKu reviewed May 25, 2023

View reviewed changes

Ziy1-Tan force-pushed the conf branch from 30f34fc to 7c23a4c Compare May 25, 2023 07:36

ilixiaocui approved these changes May 25, 2023

View reviewed changes

Ziy1-Tan force-pushed the conf branch from 7c23a4c to 78cd769 Compare May 26, 2023 02:45

Ziy1-Tan force-pushed the conf branch from 05e5e92 to 78cd769 Compare May 26, 2023 05:31

zhanghuidinah requested a review from Cyber-SiKu May 29, 2023 09:50

Ziy1-Tan force-pushed the conf branch from 78cd769 to bee6370 Compare May 30, 2023 08:17

Ziy1-Tan mentioned this pull request Jun 5, 2023

mds/metaserver: Add idle_timeout_sec option #2479

Merged

2 tasks

wuhongsong previously approved these changes Jun 6, 2023

View reviewed changes

Ziy1-Tan force-pushed the conf branch from bee6370 to 10714af Compare June 21, 2023 03:37

Ziy1-Tan force-pushed the conf branch from 10714af to d7349e3 Compare June 21, 2023 05:09

Ziy1-Tan force-pushed the conf branch from d7349e3 to 2ff9295 Compare June 22, 2023 11:05

Ziy1-Tan force-pushed the conf branch from 2ff9295 to 0453d03 Compare June 22, 2023 12:40

Ziy1-Tan force-pushed the conf branch from 0453d03 to 51f525f Compare July 6, 2023 08:54

Cyber-SiKu approved these changes Jul 6, 2023

View reviewed changes

Ziy1-Tan force-pushed the conf branch from 51f525f to 4ebed80 Compare July 22, 2023 07:54

curvefs/client: make user agent configurable

e6a5ddc

Signed-off-by: Ziy1-Tan <ajb459684460@gmail.com>

Ziy1-Tan force-pushed the conf branch from 4ebed80 to e6a5ddc Compare July 22, 2023 09:11

Ziy1-Tan requested review from wuhongsong and Cyber-SiKu July 22, 2023 11:17

Cyber-SiKu merged commit 0346d82 into opencurve:master Jul 24, 2023
3 checks passed

caoxianfei1 mentioned this pull request Jul 28, 2023

Set the UA of Aws::S3::S3Client as a configurable item #2499

Closed

caoxianfei1 mentioned this pull request Nov 15, 2023

s3 user agent configuration is incorrect. #2898

Closed

curvefs/client: make user agent configurable #2501

curvefs/client: make user agent configurable #2501

Conversation

Ziy1-Tan commented May 25, 2023

What problem does this PR solve?

What is changed and how it works?

Check List

Ziy1-Tan commented May 25, 2023

Cyber-SiKu left a comment

Choose a reason for hiding this comment

Cyber-SiKu May 25, 2023

Choose a reason for hiding this comment

Ziy1-Tan May 25, 2023

Choose a reason for hiding this comment

Cyber-SiKu May 25, 2023

Choose a reason for hiding this comment

Ziy1-Tan May 26, 2023

Choose a reason for hiding this comment

Ziy1-Tan commented May 25, 2023

Ziy1-Tan commented May 25, 2023

Ziy1-Tan commented May 25, 2023

Cyber-SiKu commented May 25, 2023

Ziy1-Tan commented May 26, 2023

Ziy1-Tan commented May 26, 2023

Ziy1-Tan commented May 26, 2023

Ziy1-Tan commented May 29, 2023

wuhongsong commented May 30, 2023

Cyber-SiKu commented May 30, 2023

Ziy1-Tan commented May 30, 2023

Ziy1-Tan commented May 30, 2023

wuhongsong commented Jun 19, 2023

SeanHai commented Jun 21, 2023

SeanHai commented Jun 21, 2023

Ziy1-Tan commented Jun 21, 2023

Ziy1-Tan commented Jun 21, 2023

Ziy1-Tan commented Jun 21, 2023

Ziy1-Tan commented Jun 21, 2023

Ziy1-Tan commented Jun 22, 2023

Ziy1-Tan commented Jun 22, 2023

Ziy1-Tan commented Jun 22, 2023

Ziy1-Tan commented Jun 24, 2023

Cyber-SiKu commented Jun 25, 2023

caoxianfei1 commented Jun 28, 2023

wuhongsong commented Jul 5, 2023

wuhongsong commented Jul 5, 2023

Ziy1-Tan commented Jul 6, 2023

SeanHai commented Jul 9, 2023

wuhongsong commented Jul 17, 2023

Ziy1-Tan commented Jul 22, 2023

Ziy1-Tan commented Jul 22, 2023

Ziy1-Tan commented Jul 22, 2023