Skip to content

[ZEPPELIN-1465] Add an option to allow S3 server-side encryption #1969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jeff-cyft wants to merge 2 commits into from
Closed

[ZEPPELIN-1465] Add an option to allow S3 server-side encryption #1969

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3c657ac
Configuration option to request S3 SSE when notebooks are saved.
jeff-cyft Feb 2, 2017
26f5264
code style - remove tab
jeff-cyft Feb 2, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions conf/zeppelin-env.cmd.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ REM set ZEPPELIN_NOTEBOOK_S3_USER REM User in bucket where notebook
REM set ZEPPELIN_NOTEBOOK_S3_ENDPOINT REM Endpoint of the bucket
REM set ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID REM AWS KMS key ID
REM set ZEPPELIN_NOTEBOOK_S3_KMS_KEY_REGION REM AWS KMS key region
REM set ZEPPELIN_NOTEBOOK_S3_SSE REM Server-side encryption enabled for notebooks
REM set ZEPPELIN_IDENT_STRING REM A string representing this instance of zeppelin. $USER by default.
REM set ZEPPELIN_NICENESS REM The scheduling priority for daemons. Defaults to 0.
REM set ZEPPELIN_INTERPRETER_LOCALREPO REM Local repository for interpreter's additional dependency loading
Expand Down
1 change: 1 addition & 0 deletions conf/zeppelin-env.sh.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
# export ZEPPELIN_NOTEBOOK_S3_USER # User in bucket where notebook saved. For example bucket/user/notebook/2A94M5J1Z/note.json
# export ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID # AWS KMS key ID
# export ZEPPELIN_NOTEBOOK_S3_KMS_KEY_REGION # AWS KMS key region
# export ZEPPELIN_NOTEBOOK_S3_SSE # Server-side encryption enabled for notebooks
# export ZEPPELIN_IDENT_STRING # A string representing this instance of zeppelin. $USER by default.
# export ZEPPELIN_NICENESS # The scheduling priority for daemons. Defaults to 0.
# export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository for interpreter's additional dependency loading
Expand Down
8 changes: 8 additions & 0 deletions conf/zeppelin-site.xml.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,14 @@
</property>
-->

<!-- Server-side encryption enabled for notebooks -->
<!--
<property>
<name>zeppelin.notebook.s3.sse</name>
<value>true</value>
<description>Server-side encryption enabled for notebooks</description>
</property>
-->

<!-- If using Azure for storage use the following settings -->
<!--
Expand Down
6 changes: 6 additions & 0 deletions docs/install/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,12 @@ If both are defined, then the **environment variables** will take priority.
<td></td>
<td>Class name of a custom S3 encryption materials provider implementation to use for encrypting data in S3 (optional)</td>
</tr>
<tr>
<td>ZEPPELIN_NOTEBOOK_S3_SSE</td>
<td>zeppelin.notebook.s3.sse</td>
<td>false</td>
<td>Save notebooks to S3 with server-side encryption enabled</td>
</tr>
<tr>
<td>ZEPPELIN_NOTEBOOK_AZURE_CONNECTION_STRING</td>
<td>zeppelin.notebook.azure.connectionString</td>
Expand Down
18 changes: 18 additions & 0 deletions docs/storage/storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,24 @@ Or using the following setting in **zeppelin-site.xml**:
<description>Custom encryption materials provider used to encrypt notebook data in S3</description>
```

#### Enable server-side encryption

To request server-side encryption of notebooks, set the following environment variable in the file **zeppelin-env.sh**:

```
export ZEPPELIN_NOTEBOOK_S3_SSE = true
```

Or using the following setting in **zeppelin-site.xml**:

```
<property>
<name>zeppelin.notebook.s3.sse</name>
<value>true</value>
<description>Server-side encryption enabled for notebooks</description>
</property>
```

</br>
## Notebook Storage in Azure <a name="Azure"></a>

Expand Down
5 changes: 5 additions & 0 deletions zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,10 @@ public String getS3EncryptionMaterialsProviderClass() {
return getString(ConfVars.ZEPPELIN_NOTEBOOK_S3_EMP);
}

public boolean isS3ServerSideEncryption() {
return getBoolean(ConfVars.ZEPPELIN_NOTEBOOK_S3_SSE);
}

public String getInterpreterListPath() {
return getRelativeDir(String.format("%s/interpreter-list", getConfDir()));
}
Expand Down Expand Up @@ -588,6 +592,7 @@ public static enum ConfVars {
ZEPPELIN_NOTEBOOK_S3_EMP("zeppelin.notebook.s3.encryptionMaterialsProvider", null),
ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID("zeppelin.notebook.s3.kmsKeyID", null),
ZEPPELIN_NOTEBOOK_S3_KMS_KEY_REGION("zeppelin.notebook.s3.kmsKeyRegion", null),
ZEPPELIN_NOTEBOOK_S3_SSE("zeppelin.notebook.s3.sse", false),
ZEPPELIN_NOTEBOOK_AZURE_CONNECTION_STRING("zeppelin.notebook.azure.connectionString", null),
ZEPPELIN_NOTEBOOK_AZURE_SHARE("zeppelin.notebook.azure.share", "zeppelin"),
ZEPPELIN_NOTEBOOK_AZURE_USER("zeppelin.notebook.azure.user", "user"),
Expand Down
15 changes: 14 additions & 1 deletion zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/S3NotebookRepo.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
import com.amazonaws.services.s3.model.KMSEncryptionMaterialsProvider;
import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
Expand Down Expand Up @@ -86,12 +87,14 @@ public class S3NotebookRepo implements NotebookRepo {
private final AmazonS3 s3client;
private final String bucketName;
private final String user;
private final boolean useServerSideEncryption;
private final ZeppelinConfiguration conf;

public S3NotebookRepo(ZeppelinConfiguration conf) throws IOException {
this.conf = conf;
bucketName = conf.getBucketName();
user = conf.getUser();
useServerSideEncryption = conf.isS3ServerSideEncryption();

// always use the default provider chain
AWSCredentialsProvider credentialsProvider = new DefaultAWSCredentialsProviderChain();
Expand Down Expand Up @@ -234,7 +237,17 @@ public void save(Note note, AuthenticationInfo subject) throws IOException {
Writer writer = new OutputStreamWriter(new FileOutputStream(file));
writer.write(json);
writer.close();
s3client.putObject(new PutObjectRequest(bucketName, key, file));

PutObjectRequest putRequest = new PutObjectRequest(bucketName, key, file);

if (useServerSideEncryption) {
// Request server-side encryption.
ObjectMetadata objectMetadata = new ObjectMetadata();
objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);
Copy link
Copy Markdown
Member

@felixcheung felixcheung Feb 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor q: should SSEAlogrithm be configurable too?

Copy link
Copy Markdown
Contributor Author

@jeff-cyft jeff-cyft Feb 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a good question.

  1. There is currently only the one documented valid option. http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/model/ObjectMetadata.html
  2. There's no enumeration in the API that can be searched by String so in the future this code would need amendment anyway to support potential new options. I thought about letting the zeppelin configurer specify any string they'd like but it seemed error-prone to expect someone configuring zeppelin to be aware of constant string literals in the S3 Java API.

Copy link
Copy Markdown
Member

@felixcheung felixcheung Feb 4, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great! thanks for the details!

putRequest.setMetadata(objectMetadata);
}

s3client.putObject(putRequest);
}
catch (AmazonClientException ace) {
throw new IOException("Unable to store note in S3: " + ace, ace);
Expand Down