New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL/TLS Support #17

Closed
okkez opened this Issue Jul 14, 2017 · 12 comments

Comments

Projects
None yet
2 participants
@okkez
Contributor

okkez commented Jul 14, 2017

It is very useful to support SSL/TLS by influent.
Because Fluentd v0.14.x in/out_forward supports SSL/TLS.
I want to use SSL/TLS in https://github.com/fluent/kafka-connect-fluentd.

I will investigate about SSL/TLS support to influent.
Do you have any idea? @okumin

@okumin

This comment has been minimized.

Show comment
Hide comment
@okumin

okumin Jul 14, 2017

Owner

@okkez I'm planning to implement all the feature defined in Forward Protocol Specification v1.
But I have no idea abount TLS now, so it is very happy if you help this project!

Owner

okumin commented Jul 14, 2017

@okkez I'm planning to implement all the feature defined in Forward Protocol Specification v1.
But I have no idea abount TLS now, so it is very happy if you help this project!

@okkez

This comment has been minimized.

Show comment
Hide comment
@okkez

okkez Jul 19, 2017

Contributor

@okumin Thank you replying! I'm investigating for this.

We need to implement NioSslAcceptor implements NioAttachment and NioSslChannel implements AutoClosable. And then we can add related parameters to NioForwardServer.
Is this right? If wrong, could you help me?

FYI: Sample code TLS server client https://github.com/alkarn/sslengine.example

Contributor

okkez commented Jul 19, 2017

@okumin Thank you replying! I'm investigating for this.

We need to implement NioSslAcceptor implements NioAttachment and NioSslChannel implements AutoClosable. And then we can add related parameters to NioForwardServer.
Is this right? If wrong, could you help me?

FYI: Sample code TLS server client https://github.com/alkarn/sslengine.example

@okkez

This comment has been minimized.

Show comment
Hide comment
@okkez
Contributor

okkez commented Jul 21, 2017

@okumin

This comment has been minimized.

Show comment
Hide comment
@okumin

okumin Jul 22, 2017

Owner

@okkez Looks good!
Are you going to complete this work?
If so, I hope for you to PR!
I'm learning TLS on nio and waiting that.

Now Influent's test codes are implemented in Scala, so I don't require you to write tests.

Owner

okumin commented Jul 22, 2017

@okkez Looks good!
Are you going to complete this work?
If so, I hope for you to PR!
I'm learning TLS on nio and waiting that.

Now Influent's test codes are implemented in Scala, so I don't require you to write tests.

@okkez okkez referenced this issue Jul 22, 2017

Closed

wip: Suport tls #22

@okkez

This comment has been minimized.

Show comment
Hide comment
@okkez

okkez Jul 22, 2017

Contributor

Are you going to complete this work?

I want to complete this work in several days.
I've created new PR #22, but it is work in progress.
@okumin Thank you for your offer. Feel free to comment or create PRs.
I'm not Java expert so I want your comment and PRs.

I've trying to run TLS support with Fluentd v0.14.x, but it has not work yet...
I tried to create keys and certs using following command (not work):

$ keytool -genkey -keystore ./keystore.jks -storepass password -alias fluentd
$ keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.pkcs12 -srcstoretype jks -deststoretype pkcs12
$ openssl pkcs12 -in keystore.pkcs12 -clcerts -nokeys -nodes -out keystore_cert.pem
$ openssl pkcs12 -in keystore.pkcs12 -nocerts -nodes -out keystore_key.pem
Contributor

okkez commented Jul 22, 2017

Are you going to complete this work?

I want to complete this work in several days.
I've created new PR #22, but it is work in progress.
@okumin Thank you for your offer. Feel free to comment or create PRs.
I'm not Java expert so I want your comment and PRs.

I've trying to run TLS support with Fluentd v0.14.x, but it has not work yet...
I tried to create keys and certs using following command (not work):

$ keytool -genkey -keystore ./keystore.jks -storepass password -alias fluentd
$ keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.pkcs12 -srcstoretype jks -deststoretype pkcs12
$ openssl pkcs12 -in keystore.pkcs12 -clcerts -nokeys -nodes -out keystore_cert.pem
$ openssl pkcs12 -in keystore.pkcs12 -nocerts -nodes -out keystore_key.pem
@okumin

This comment has been minimized.

Show comment
Hide comment
@okumin

okumin Jul 22, 2017

Owner

OK, I wrote some comments.

I have read some documents and alkarn/sslengine.example) and I'm feeling some problems.

  1. Implementing SSL/TLS functions to (mainly) influent-transport
  2. Providing SSL/TLS configuration API to ForwardServer.Builder
  3. Performance optimizations

The first, I want to focus 1 and 2.
Since SSL/TLS requires some extra buffer and memory copy, it may be possible to optimize.
I have a plan to resolve such a problem at #14.

But only thread-blocking has big impact to performance and API design and should not be compromised.

Owner

okumin commented Jul 22, 2017

OK, I wrote some comments.

I have read some documents and alkarn/sslengine.example) and I'm feeling some problems.

  1. Implementing SSL/TLS functions to (mainly) influent-transport
  2. Providing SSL/TLS configuration API to ForwardServer.Builder
  3. Performance optimizations

The first, I want to focus 1 and 2.
Since SSL/TLS requires some extra buffer and memory copy, it may be possible to optimize.
I have a plan to resolve such a problem at #14.

But only thread-blocking has big impact to performance and API design and should not be compromised.

@okumin

This comment has been minimized.

Show comment
Hide comment
@okumin

okumin Jul 22, 2017

Owner

I've trying to run TLS support with Fluentd v0.14.x, but it has not work yet...

I'll try that.

Owner

okumin commented Jul 22, 2017

I've trying to run TLS support with Fluentd v0.14.x, but it has not work yet...

I'll try that.

@okumin

This comment has been minimized.

Show comment
Hide comment
@okumin

okumin Aug 6, 2017

Owner

@okkez I published .
This version provides SSL/TLS support.
But notice that 0.4.0-M1 is not stable and TLS implementation has bugs and performance issues with very high probability.
This implementation is built on top of the previous structure and not good yet.

Usage

1. Generate private CA

Execute bin/generate-ca.sh.
out/ca_key.pem and out/ca_cert.pem are generated.

./bin/generate-ca.sh

2. Generate a key store for Influent

Execute bin/generate-server-keystore.sh.
Note that server's common name should be the host name(localhost is useful if client coexists with server).
out/influent-server.jks is generated.

./bin/generate-server-keystore.sh

3. Configure fluentd

tls_cert_path is required.

<source>
  @type dummy
  tag mofu
  dummy {"hello":"world"}
</source>

<match *>
  @type forward
  flush_interval 1s
  require_ack_response true

  transport tls
  tls_cert_path /path/to/ca_cert.pem

  <server>
    name localhost
    host localhost
    port 24224
  </server>
</match>

4. Start server

sbt "project influentJavaSample" "runMain sample.TLSPrint 1000"

5. Start client

fluentd -c /path/to/fluent.conf
Owner

okumin commented Aug 6, 2017

@okkez I published .
This version provides SSL/TLS support.
But notice that 0.4.0-M1 is not stable and TLS implementation has bugs and performance issues with very high probability.
This implementation is built on top of the previous structure and not good yet.

Usage

1. Generate private CA

Execute bin/generate-ca.sh.
out/ca_key.pem and out/ca_cert.pem are generated.

./bin/generate-ca.sh

2. Generate a key store for Influent

Execute bin/generate-server-keystore.sh.
Note that server's common name should be the host name(localhost is useful if client coexists with server).
out/influent-server.jks is generated.

./bin/generate-server-keystore.sh

3. Configure fluentd

tls_cert_path is required.

<source>
  @type dummy
  tag mofu
  dummy {"hello":"world"}
</source>

<match *>
  @type forward
  flush_interval 1s
  require_ack_response true

  transport tls
  tls_cert_path /path/to/ca_cert.pem

  <server>
    name localhost
    host localhost
    port 24224
  </server>
</match>

4. Start server

sbt "project influentJavaSample" "runMain sample.TLSPrint 1000"

5. Start client

fluentd -c /path/to/fluent.conf
@okumin

This comment has been minimized.

Show comment
Hide comment
@okumin

okumin Aug 6, 2017

Owner

I have upgraded kafka-connect-fluentd.
fluent/kafka-connect-fluentd#4

I'm sorry I'm late.

Owner

okumin commented Aug 6, 2017

I have upgraded kafka-connect-fluentd.
fluent/kafka-connect-fluentd#4

I'm sorry I'm late.

@okkez

This comment has been minimized.

Show comment
Hide comment
@okkez

okkez Aug 7, 2017

Contributor

Thanks! I will test SSL/TLS support.

Contributor

okkez commented Aug 7, 2017

Thanks! I will test SSL/TLS support.

@okkez

This comment has been minimized.

Show comment
Hide comment
@okkez

okkez Aug 8, 2017

Contributor

It works on my local environment.

Contributor

okkez commented Aug 8, 2017

It works on my local environment.

@okumin

This comment has been minimized.

Show comment
Hide comment
@okumin

okumin Aug 8, 2017

Owner

Thanks for your confirmation!

Owner

okumin commented Aug 8, 2017

Thanks for your confirmation!

@okumin okumin closed this Aug 8, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment