How to properly debug logging in multi-go-routines environment to allow restore request->response sequence?

[Slach]

I have S3 client
with following code

type S3LogToZeroLogAdapter struct {
	logger zerolog.Logger
}

func newS3Logger(logger zerolog.Logger) S3LogToZeroLogAdapter {
	return S3LogToZeroLogAdapter{
		logger: logger,
	}
}

func (S3LogToApexLogAdapter S3LogToZeroLogAdapter) Logf(severity awsV2Logging.Classification, msg string, args ...interface{}) {
	msg = fmt.Sprintf("[s3:%s] %s", severity, msg)
	if len(args) > 0 {
		S3LogToApexLogAdapter.logger.Info().Msgf(msg, args...)
	} else {
		S3LogToApexLogAdapter.logger.Info().Msg(msg)
	}
}
...
func (s *S3) Connect(ctx context.Context) error {
	var awsConfig aws.Config
	awsConfig, err = awsV2Config.LoadDefaultConfig(
		ctx,
		awsV2Config.WithRetryMode(aws.RetryModeAdaptive),
	)
        awsConfig.Logger = newS3Logger(log.Logger)
	awsConfig.ClientLogMode = aws.LogRetries | aws.LogRequest | aws.LogResponse
...
}

in logs i see following picture

2025-01-21 14:04:52.288 INF pkg/storage/s3.go:49 > [s3:DEBUG] Request
PUT /clickhouse/test1/shadow/datac_ods/table/default_all_2_2_0_50.tar?partNumber=1&uploadId=M2MyMGQyODQtNzJhNS00OTM4LTk2ZjEtOTBiYjhmNTU2NWJmLmU0ZmI3ZDYzLTgxN2MtNDBmZi1iNTI4LTkwZjAzMzUxZTg2MA&x-id=UploadPart HTTP/1.1
Host: 192.168.12.41:8000
User-Agent: m/F,G aws-sdk-go-v2/1.32.2 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.66.0 ft/s3-transfer
Content-Length: 14310987
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 87836c66-d4a3-4753-9b61-40fe9f2c7410
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=*******************/20250121/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=b3e04a8bfe96606964e4c68d06d1e1a4cf9dddf7ac279462b246d2fda6e475c4
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Content-Sha256: ce33b93b8e2446f1e9c889558f80c1f467f2dad5dff797f82f8e5ed2ce193801
X-Amz-Date: 20250121T060452Z


2025-01-21 14:04:52.290 INF pkg/storage/s3.go:49 > [s3:DEBUG] Request
PUT /clickhouse/test1/shadow/datac_ods/table/default_all_3_3_0_50.tar?partNumber=1&uploadId=M2MyMGQyODQtNzJhNS00OTM4LTk2ZjEtOTBiYjhmNTU2NWJmLjViMGFhZGFlLWM0YzItNGY0Mi05Y2YyLTcxMWJkZTVjM2I5ZA&x-id=UploadPart HTTP/1.1
Host: 192.168.12.41:8000
User-Agent: m/F,G aws-sdk-go-v2/1.32.2 os/linux lang/go#1.23.2 md/GOOS#linux md/GOARCH#amd64 api/s3#1.66.0 ft/s3-transfer
Content-Length: 14310987
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 3c6e2fe6-31cd-440e-99d4-409f8aa54322
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=*******************/20250121/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=7edb4811c979a06aedfec81e0d128f11e08bbd8859708c042f1eacfc898d0b61
Content-Type: application/octet-stream
Expect: 100-continue
X-Amz-Content-Sha256: 57513c1a75ee22ecb4be444d1387fb4efa830d9ff824671a3bb70a9d73fc7014
X-Amz-Date: 20250121T060452Z


2025-01-21 14:06:56.308 INF pkg/storage/s3.go:49 > [s3:DEBUG] Response
HTTP/1.1 400 Bad Request
Content-Length: 503
Accept-Ranges: bytes
Content-Type: application/xml
Date: Tue, 21 Jan 2025 06:06:08 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 03f6d7ba09b0531a178059659f12e65ab6a75adddf2f548b1f37624d55d95fba
X-Amz-Request-Id: 181CA00B843BD44D
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

I don't understand which Request is related to Response, default_all_3_3_0_50 or default_all_2_2_0_50

How to properly initialize logger to link response and request?

[codelinx]

Hi,

• can you show the complete code?
• can you show an example of the output you would like to see?

[Slach]

@codelinx sorry
we already have
#2973

just adding invoke id into logs and separate constant for it
will be enough

something like aws.LogInvokeId and
awsConfig.ClientLogMode = aws.LogInvokeId | aws.LogRetries | aws.LogRequest | aws.LogResponse

and output will be something like that

2025-01-21 14:04:52.288 INF pkg/storage/s3.go:49 > [s3:DEBUG] Request
Invoke-Id: XXXXXXXXXXXXXXXXXXXXX
PUT /clickhouse/test1/shadow/datac_ods/table/default_all_2_2_0_50.tar?partNumber=1&uploadId=M2MyMGQyODQtNzJhNS00OTM4LTk2ZjEtOTBiYjhmNTU2NWJmLmU0ZmI3ZDYzLTgxN2MtNDBmZi1iNTI4LTkwZjAzMzUxZTg2MA&x-id=UploadPart HTTP/1.1
...


2025-01-21 14:06:56.308 INF pkg/storage/s3.go:49 > [s3:DEBUG] Response
Invoke-Id: XXXXXXXXXXXXXXXXXXXXX
HTTP/1.1 400 Bad Request
Content-Length: 503