Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-2140. Add robot test for GDPR feature #1542

Closed
wants to merge 2 commits into from

Conversation

dineshchitlangia
Copy link
Contributor

@dineshchitlangia dineshchitlangia commented Sep 28, 2019

Tested using test-single script.

$ ../test-single.sh om gdpr/gdpr.robot
================================================
ozone-gdpr :: Smoketest Ozone GDPR Feature
================================================
Test GDPR(disabled) without explicit options                       | PASS |
------------------------------------------------------------------------------
Test GDPR with --enforcegdpr=true                                     | PASS |
------------------------------------------------------------------------------
Test GDPR with -g=true                                                         | PASS |
------------------------------------------------------------------------------
Test GDPR with -g=false                                                       | PASS |
------------------------------------------------------------------------------
ozone-gdpr :: Smoketest Ozone GDPR Feature                   | PASS |
4 critical tests, 4 passed, 0 failed
4 tests total, 4 passed, 0 failed
============================================================
Output:  /tmp/smoketest/ozone/result/robot-ozone-ozone-gdpr-om.xml
Log:     ~/apache/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone/result/log.html
Report:  ~/apache/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone/result/report.html

@dineshchitlangia
Copy link
Contributor Author

/label ozone

@elek elek added the ozone label Sep 28, 2019
@dineshchitlangia
Copy link
Contributor Author

@elek Request your review pls. Thanks!

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 1727 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 mvninstall 30 hadoop-hdds in trunk failed.
-1 mvninstall 38 hadoop-ozone in trunk failed.
-1 compile 18 hadoop-hdds in trunk failed.
-1 compile 13 hadoop-ozone in trunk failed.
+1 mvnsite 0 trunk passed
+1 shadedclient 949 branch has no errors when building and testing our client artifacts.
-1 javadoc 18 hadoop-hdds in trunk failed.
-1 javadoc 16 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
-1 mvninstall 30 hadoop-hdds in the patch failed.
-1 mvninstall 34 hadoop-ozone in the patch failed.
-1 compile 20 hadoop-hdds in the patch failed.
-1 compile 15 hadoop-ozone in the patch failed.
-1 javac 20 hadoop-hdds in the patch failed.
-1 javac 15 hadoop-ozone in the patch failed.
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 777 patch has no errors when building and testing our client artifacts.
-1 javadoc 18 hadoop-hdds in the patch failed.
-1 javadoc 16 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 23 hadoop-hdds in the patch failed.
-1 unit 22 hadoop-ozone in the patch failed.
+1 asflicense 29 The patch does not generate ASF License warnings.
3845
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/Dockerfile
GITHUB PR #1542
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient
uname Linux c181086e2738 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 14b4fbc
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-javadoc-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/testReport/
Max. process+thread count 305 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/1/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@dineshchitlangia
Copy link
Contributor Author

Failures are unrelated to the patch.
Checkstyle issue is addressed by HDDS-2202

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dineshchitlangia the patch. Overall it looks good to me I have one comment about the impotency.

Unrelated to this patch (as this patch tests the CLI arguments) but I am wondering how the core GDPR feature can be tested. I mean how can we be sure that the data is really unreadable (grep to the chunk files for a specific strings??). To be honest, I have no idea, but putting this interesting question to here ;-)


*** Test Cases ***
Test GDPR(disabled) without explicit options
Execute ozone sh volume create /${volume} --quota 100TB
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually I prefer to make the robot tests idempotent. I think it's better to support to run them multiple times and it more safe if we use brand new volume all the time. In ozone-shell it's solved by adding a random string to all the volume names.

*** Variables ***
${prefix}    generated

*** Keywords ***
Generate prefix
   ${random} =         Generate Random String  5  [NUMBERS]
   Set Suite Variable  ${prefix}  ${random}

*** Test Cases ***
RpcClient with port
   Test ozone shell       o3://            om:9862     ${prefix}-rpcwoport

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduced random volume name in recent commit.

@dineshchitlangia
Copy link
Contributor Author

Unrelated to this patch (as this patch tests the CLI arguments) but I am wondering how the core GDPR feature can be tested. I mean how can we be sure that the data is really unreadable (grep to the chunk files for a specific strings??). To be honest, I have no idea, but putting this interesting question to here ;-)

Recap: GDPR talk in Vegas ;)

  • When putting a key in a GDPR enforced bucket, Ozone will create a symmetric key and Client will use that to encrypt and write to key.
  • This encryption key is stored in KeyInfo Metadata
  • When reading the key, the encryption key is fetched from KeyInfo Metadata and used to decrypt the key.

After our Vegas conference, we modified the delete path (HDDS-2174):

  • When user asks Ozone to delete a Key, we first delete the encryption key details from KeyInfo Metadata, then we move the KeyInfo to DeletedTable in OM.
  • Since the encryption key is lost, there is no way you can read that data(except if you restore a back/snapshot of your entire system from before deletion, which will also be address in version 2)
  • HDDS-2174 included a test to confirm the key metadata in DeletedTable does not have the GDPR Encryption Key details. Thereby, even if you get your hands on chunks, you will still read encrypted junk :)

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 38 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
-1 mvninstall 40 hadoop-hdds in trunk failed.
-1 mvninstall 41 hadoop-ozone in trunk failed.
-1 compile 21 hadoop-hdds in trunk failed.
-1 compile 15 hadoop-ozone in trunk failed.
+1 mvnsite 0 trunk passed
+1 shadedclient 887 branch has no errors when building and testing our client artifacts.
-1 javadoc 22 hadoop-hdds in trunk failed.
-1 javadoc 20 hadoop-ozone in trunk failed.
_ Patch Compile Tests _
-1 mvninstall 32 hadoop-hdds in the patch failed.
-1 mvninstall 38 hadoop-ozone in the patch failed.
-1 compile 25 hadoop-hdds in the patch failed.
-1 compile 19 hadoop-ozone in the patch failed.
-1 javac 25 hadoop-hdds in the patch failed.
-1 javac 19 hadoop-ozone in the patch failed.
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 711 patch has no errors when building and testing our client artifacts.
-1 javadoc 22 hadoop-hdds in the patch failed.
-1 javadoc 20 hadoop-ozone in the patch failed.
_ Other Tests _
-1 unit 27 hadoop-hdds in the patch failed.
-1 unit 25 hadoop-ozone in the patch failed.
+1 asflicense 34 The patch does not generate ASF License warnings.
2087
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/Dockerfile
GITHUB PR #1542
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient
uname Linux 85c3581520f0 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / c99a121
Default Java 1.8.0_222
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/branch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/branch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/branch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/branch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/branch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/branch-javadoc-hadoop-ozone.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-mvninstall-hadoop-hdds.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-mvninstall-hadoop-ozone.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-compile-hadoop-hdds.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-compile-hadoop-ozone.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-compile-hadoop-hdds.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-compile-hadoop-ozone.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-javadoc-hadoop-hdds.txt
javadoc https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-javadoc-hadoop-ozone.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/testReport/
Max. process+thread count 400 (vs. ulimit of 5500)
modules C: hadoop-ozone/dist U: hadoop-ozone/dist
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1542/2/console
versions git=2.7.4 maven=3.3.9
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@elek
Copy link
Member

elek commented Oct 4, 2019

Thereby, even if you get your hands on chunks, you will still read encrypted junk :)

Yes, I understand. That's the question. How can we prove that we only have junk. I would like to prove that the junk is encrypted.

Copy link
Member

@elek elek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 Thanks the update @dineshchitlangia

Tested (twice!) and worked well.

One thing what I found, I think it's not yet enabled in the daily builds.

I think in the hadoop-ozone/dist/src/main/compose/ozone/test.sh we need a new line:

execute_robot_test gdpr.robot

But let's do it in a follow-up jira. To many issues in the queue. I will commit it right now...

@elek elek closed this in d061c84 Oct 4, 2019
@dineshchitlangia
Copy link
Contributor Author

Thereby, even if you get your hands on chunks, you will still read encrypted junk :)

Yes, I understand. That's the question. How can we prove that we only have junk. I would like to prove that the junk is encrypted.

@elek Such a test exists in UT at TestOzoneRpcClientAbstract#testKeyReadWriteForGDPR

@elek
Copy link
Member

elek commented Oct 4, 2019

@elek Such a test exists in UT at TestOzoneRpcClientAbstract#testKeyReadWriteForGDPR

Wow, and with perfect java description. Yes, it's exactly that. It's my shame that I didn't know.

You deserve a photo with this source code:
http://www.peteradamsphoto.com/apache-20th-anniversary/

;-)

@dineshchitlangia
Copy link
Contributor Author

@elek Such a test exists in UT at TestOzoneRpcClientAbstract#testKeyReadWriteForGDPR

Wow, and with perfect java description. Yes, it's exactly that. It's my shame that I didn't know.

You deserve a photo with this source code:
http://www.peteradamsphoto.com/apache-20th-anniversary/

;-)

Thanks, because it was a complex test and I wanted to ensure that no one accidentally modifies the test without getting the context, I made it so descriptive. Glad it looks good and helps.

Good Idea, I will get a pic like that 😄

@dineshchitlangia
Copy link
Contributor Author

But let's do it in a follow-up jira. To many issues in the queue. I will commit it right now...

Addressed this in HDDS-2252

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants