Encryption support ext consumer #612

GPrabhudas · 2021-09-06T08:18:48Z

Breakdown of PR #552

This PR includes the encryption/decryption changes at consumer side.

- use base crypto package for encryption

- move it to Consumer MR

…Fanatics/pulsar-client-go into encryption-support-ext-producer

…t-ext-consumer

- move decryptor creation to partition_producer.go - update reader_impl - update consumer_impl

cckellogg · 2021-09-07T19:31:25Z

pulsar/impl_message.go

@@ -215,6 +233,7 @@ type message struct {
 	replicatedFrom      string
 	redeliveryCount     uint32
 	schema              Schema
+	encryptionContext   EncryptionContext


Let's make this a pointer

cckellogg · 2021-09-07T19:37:05Z

pulsar/internal/crypto/decryptor.go

+// Decryptor support decrypting of message
+type Decryptor interface {
+	Decrypt(payload []byte, msgID *pb.MessageIdData, msgMetadata *pb.MessageMetadata) ([]byte, error)
+	CryptoFailureAction() int


Why is this needed? I think the Decryptor should just have Decrypt and return an error. The caller of this can then figure out what to do with the error.

cckellogg · 2021-09-07T19:38:54Z

pulsar/message.go

+
+	// GetEncryptionContext get the ecryption context of message
+	// It will be used by the application to parse undecrypted message
+	GetEncryptionContext() EncryptionContext


Let's return a pointer or interface

cckellogg · 2021-09-07T19:41:47Z

pulsar/consumer_partition.go

+		pc.log.Error(err)
+		switch pc.decryptor.CryptoFailureAction() {
+		case crypto.ConsumerCryptoFailureActionFail:
+			pc.log.Errorf("consuming message failed due to decryption err :%v", err)


The java clients add this to the unacked message tracker do we need to do the same?

Let me take a look at this..

I don't think we have this so may we just need to ack the message so it's not resent?

yeah, but i don't think it is good idea to do ack.

Let's say user wants to consume this message again by providing proper crypto configuration, if we do ack, then he may not be able to consume.

So, I think it's better to resend the message than doing ack.
Any other thoughts here ??

Yes, we can do nack. I'll push changes with nack on failure.

@cckellogg I've made the changes

Looks good to me will you please fix the CI issue.

thanks @cckellogg

I don't think CI is failing due to last commit.

I see most of the open PR's are failing with the same issue as below.

TestNamespaceTopicsNamespaceDoesNotExit (56.25s) client_impl_test.go:387: Error Trace: client_impl_test.go:387 Error: Expected nil, but got: &errors.errorString{s:"server error: AuthorizationError: Exception occurred while trying to authorize GetTopicsOfNamespace"} Test: TestNamespaceTopicsNamespaceDoesNotExit

And last commit do not have any changes that effects client_impl.go. More likely this has something to do with broker config that CI uses for running test cases.

please let me know If I'm wrong here.

@cckellogg looks like the CI is passed now.
Please do suggest if there are any other improvements on this PR.

cckellogg · 2021-09-07T19:55:04Z

pulsar/consumer_partition.go

+
+	// error decrypting the payload
+	if err != nil {
+		pc.log.Error(err)


remove this and add a log under ConsumerCryptoFailureActionDiscard so there is more context.

cckellogg · 2021-09-07T19:58:19Z

pulsar/consumer_partition.go

+				replicationClusters: msgMeta.GetReplicateTo(),
+				replicatedFrom:      msgMeta.GetReplicatedFrom(),
+				redeliveryCount:     response.GetRedeliveryCount(),
+				encryptionContext:   createEncryptionContext(msgMeta),


Does the encryptionContext need to be added to messages that don't fail?

I don't think there is a need to add this.

So the encryption context is only needed for failed messages?

cckellogg · 2021-09-07T20:00:24Z

pulsar/consumer_partition.go

+		case crypto.ConsumerCryptoFailureActionConsume:
+			pc.log.Warnf("consuming encrypted message due to error in decryption :%v", err)
+			messages = append(messages, &message{
+				publishTime:  timeFromUnixTimestampMillis(msgMeta.GetPublishTime()),


Nit. we can use create a new variable here and move messages := make([]*message, 0) back to where it was

messages := []*message{ { }, }

PGarule and others added 22 commits July 7, 2021 23:02

add ability to encrypt messages

d8bb8fe

- use base crypto package for encryption

fix typo

aa40a8d

lint fixes

3b8d912

address review suggestions

e1a8108

revert go mod

d200b7c

remove encryption context

feaf120

- move it to Consumer MR

try to fix check issues

a2bd72b

Merge branch 'master' into encryption-support-ext-producer

f204347

remove unused code

a561923

Merge branch 'encryption-support-ext-producer' of https://github.com/…

559dbaf

…Fanatics/pulsar-client-go into encryption-support-ext-producer

consumer encryption/decryption changes

f0ab398

remove embedded crypto struct

d7246bc

Merge branch 'encryption-support-ext-producer' into encryption-suppor…

f9c23e9

…t-ext-consumer

remove embedded struct

3939030

add comments

2ab2e20

Merge branch 'master' into encryption-support-ext-consumer

b342a8d

merge conflict issues fix

134ae2e

add noop decryptor

f27838f

lint fixes

b296c27

fix test case

56915e8

refactor and reader encryption changes

6012ae1

refactor

85622e6

- move decryptor creation to partition_producer.go - update reader_impl - update consumer_impl

GPrabhudas mentioned this pull request Sep 7, 2021

Encryption support in Go client #552

Closed

cckellogg reviewed Sep 7, 2021

View reviewed changes

PGarule added 3 commits September 8, 2021 12:15

address review feedback

1fea522

Nack on decryption failure

f35c3c9

add comment on test case

f441348

cckellogg approved these changes Sep 28, 2021

View reviewed changes

cckellogg requested a review from wolfstudy October 6, 2021 14:24

wolfstudy assigned GPrabhudas Oct 9, 2021

wolfstudy added this to the 0.7.0 milestone Oct 9, 2021

wolfstudy approved these changes Oct 9, 2021

View reviewed changes

wolfstudy merged commit ab96ad7 into apache:master Oct 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encryption support ext consumer #612

Encryption support ext consumer #612

GPrabhudas commented Sep 6, 2021

cckellogg Sep 7, 2021

GPrabhudas Sep 8, 2021

cckellogg Sep 7, 2021

GPrabhudas Sep 8, 2021

cckellogg Sep 7, 2021

GPrabhudas Sep 8, 2021

cckellogg Sep 7, 2021

GPrabhudas Sep 8, 2021

cckellogg Sep 8, 2021

GPrabhudas Sep 8, 2021

GPrabhudas Sep 8, 2021

GPrabhudas Sep 21, 2021

GPrabhudas Sep 21, 2021

cckellogg Sep 22, 2021

GPrabhudas Sep 23, 2021

GPrabhudas Sep 28, 2021

cckellogg Sep 7, 2021

GPrabhudas Sep 8, 2021

cckellogg Sep 7, 2021

GPrabhudas Sep 8, 2021 •

edited

Loading

cckellogg Sep 8, 2021

GPrabhudas Sep 8, 2021

cckellogg Sep 7, 2021

GPrabhudas Sep 8, 2021

Encryption support ext consumer #612

Encryption support ext consumer #612

Conversation

GPrabhudas commented Sep 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GPrabhudas Sep 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GPrabhudas Sep 8, 2021 •

edited

Loading