Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue-76] update samples to new builder API #79

Merged
merged 3 commits into from
May 17, 2018

Conversation

EronWright
Copy link
Contributor

Change Log

  • Added Utils class to create streams as needed
  • Updates to use builder syntax
  • Remove explicit scale policy from word count, anomaly detection samples

How to Test

  • Manual testing in Flink cluster

Signed-off-by: Eron Wright <eronwright@gmail.com>
@EronWright
Copy link
Contributor Author

I opened this PR as a heads-up, but frankly I am still testing the samples so I wouldn't recommend merging yet.

Signed-off-by: Eron Wright <eronwright@gmail.com>
LOG.info("Succesfully created stream: {}", streamId);
Stream streamId = getStreamId();

try(StreamManager streamManager = StreamManager.create(pravegaConfig.getClientConfig())) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we are duplicating Utils.createStream() here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was unable to reuse Utils here because it is in the flink-examples project and so there's way to use it from the anomaly-detection project. Personally I don't think it important to consolidate it.

@EronWright
Copy link
Contributor Author

Happy to report that the anomaly detector is working:

Event: 2018-05-17T02:32:11.360Z: 29.142.147.168: b
Event: 2018-05-17T02:32:13.361Z: 90.40.50.102: c
Event: 2018-05-17T02:32:15.362Z: 87.133.23.156: a
Event: 2018-05-17T02:32:17.363Z: 40.105.41.107: c
Event: 2018-05-17T02:32:19.364Z: 105.188.245.18: c
Event: 2018-05-17T02:32:21.365Z: 120.247.116.124: d
Event: 2018-05-17T02:32:23.366Z: 105.153.123.124: b
Event: 2018-05-17T02:32:25.367Z: 85.120.104.88: c
Event: 2018-05-17T02:32:27.368Z: 100.55.79.60: c
Event: 2018-05-17T02:32:29.369Z: 47.5.114.210: d
Event: 2018-05-17T02:32:31.370Z: 0.92.122.104: c
Event: 2018-05-17T02:32:33.371Z: 81.123.159.46: a
Event: 2018-05-17T02:32:35.372Z: 76.2.181.87: a
ALERT: 2: 2018-05-17T02:32:36.151Z: 51.142.242.229: X -&gt; g EVENT = {Event: 2018-05-17T02:29:21.209Z: 51.142.242.229: g}
ALERT: 3: 2018-05-17T02:32:36.259Z: 40.150.162.127: X -&gt; c EVENT = {Event: 2018-05-17T02:29:43.224Z: 40.150.162.127: c}
ALERT: 4: 2018-05-17T02:32:36.261Z: 74.15.162.71: W -&gt; c EVENT = {Event: 2018-05-17T02:29:45.227Z: 74.15.162.71: c}
ALERT: 2: 2018-05-17T02:32:36.263Z: 126.71.202.203: W -&gt; ` EVENT = {Event: 2018-05-17T02:29:47.228Z: 126.71.202.203: `}
ALERT: 3: 2018-05-17T02:32:36.265Z: 69.162.32.88: W -&gt; ` EVENT = {Event: 2018-05-17T02:30:07.241Z: 69.162.32.88: `}
ALERT: 3: 2018-05-17T02:32:36.267Z: 40.77.196.38: X -&gt; ` EVENT = {Event: 2018-05-17T02:30:17.247Z: 40.77.196.38: `}
ALERT: 3: 2018-05-17T02:32:36.268Z: 110.65.146.117: W -&gt; ` EVENT = {Event: 2018-05-17T02:30:25.253Z: 110.65.146.117: `}
ALERT: 3: 2018-05-17T02:32:36.269Z: 25.231.9.1: W -&gt; d EVENT = {Event: 2018-05-17T02:30:33.260Z: 25.231.9.1: d}
ALERT: 4: 2018-05-17T02:32:36.270Z: 111.238.17.97: X -&gt; f EVENT = {Event: 2018-05-17T02:30:41.264Z: 111.238.17.97: f}
ALERT: 1: 2018-05-17T02:32:36.271Z: 123.210.139.135: W -&gt; g EVENT = {Event: 2018-05-17T02:30:43.265Z: 123.210.139.135: g}
ALERT: 1: 2018-05-17T02:32:36.272Z: 23.156.120.161: X -&gt; f EVENT = {Event: 2018-05-17T02:30:51.270Z: 23.156.120.161: f}
ALERT: 3: 2018-05-17T02:32:36.273Z: 63.100.70.113: W -&gt; c EVENT = {Event: 2018-05-17T02:30:53.271Z: 63.100.70.113: c}
ALERT: 4: 2018-05-17T02:32:36.274Z: 125.169.117.40: W -&gt; f EVENT = {Event: 2018-05-17T02:31:01.276Z: 125.169.117.40: f}
ALERT: 2: 2018-05-17T02:32:36.275Z: 4.251.194.212: X -&gt; g EVENT = {Event: 2018-05-17T02:31:07.280Z: 4.251.194.212: g}
ALERT: 2: 2018-05-17T02:32:36.276Z: 74.85.79.113: X -&gt; e EVENT = {Event: 2018-05-17T02:31:15.284Z: 74.85.79.113: e}
ALERT: 1: 2018-05-17T02:32:36.278Z: 49.106.227.186: W -&gt; g EVENT = {Event: 2018-05-17T02:31:21.288Z: 49.106.227.186: g}
ALERT: 2: 2018-05-17T02:32:36.279Z: 84.97.146.31: W -&gt; c EVENT = {Event: 2018-05-17T02:31:49.307Z: 84.97.146.31: c}
ALERT: 1: 2018-05-17T02:32:36.280Z: 126.34.159.25: X -&gt; a EVENT = {Event: 2018-05-17T02:32:01.313Z: 126.34.159.25: a}
ALERT: 4: 2018-05-17T02:32:36.281Z: 102.56.86.77: W -&gt; c EVENT = {Event: 2018-05-17T02:32:09.358Z: 102.56.86.77: c}
ALERT: 4: 2018-05-17T02:32:36.349Z: 90.40.50.102: W -&gt; c EVENT = {Event: 2018-05-17T02:32:13.361Z: 90.40.50.102: c}
ALERT: 1: 2018-05-17T02:32:36.351Z: 47.5.114.210: W -&gt; d EVENT = {Event: 2018-05-17T02:32:29.369Z: 47.5.114.210: d}
ALERT: 3: 2018-05-17T02:32:36.352Z: 81.123.159.46: W -&gt; a EVENT = {Event: 2018-05-17T02:32:33.371Z: 81.123.159.46: a}
Summary: Network ID: 2 Hosts: [51.142.242.229] Total: 1 Start Time: [2018-05-17T02:29:21.209Z] End Time: [2018-05-17T02:29:21.209Z] Location: [40.7127837,-74.0059413] 
Summary: Network ID: 3 Hosts: [40.150.162.127] Total: 1 Start Time: [2018-05-17T02:29:43.224Z] End Time: [2018-05-17T02:29:43.224Z] Location: [51.5073509,-0.1277583] 
Summary: Network ID: 2 Hosts: [126.71.202.203] Total: 1 Start Time: [2018-05-17T02:29:47.228Z] End Time: [2018-05-17T02:29:47.228Z] Location: [40.7127837,-74.0059413] 
Summary: Network ID: 4 Hosts: [74.15.162.71] Total: 1 Start Time: [2018-05-17T02:29:45.227Z] End Time: [2018-05-17T02:29:45.227Z] Location: [35.7090259,139.73199249999993] 
Summary: Network ID: 3 Hosts: [69.162.32.88, 40.77.196.38, 110.65.146.117] Total: 3 Start Time: [2018-05-17T02:30:07.241Z] End Time: [2018-05-17T02:30:25.253Z] Location: [51.5073509,-0.1277583] 
Summary: Network ID: 3 Hosts: [25.231.9.1, 63.100.70.113] Total: 2 Start Time: [2018-05-17T02:30:33.260Z] End Time: [2018-05-17T02:30:53.271Z] Location: [51.5073509,-0.1277583] 
Summary: Network ID: 4 Hosts: [111.238.17.97] Total: 1 Start Time: [2018-05-17T02:30:41.264Z] End Time: [2018-05-17T02:30:41.264Z] Location: [35.7090259,139.73199249999993] 
Summary: Network ID: 1 Hosts: [123.210.139.135, 23.156.120.161] Total: 2 Start Time: [2018-05-17T02:30:43.265Z] End Time: [2018-05-17T02:30:51.270Z] Location: [37.3860517,-122.0838511] 
Summary: Network ID: 4 Hosts: [125.169.117.40] Total: 1 Start Time: [2018-05-17T02:31:01.276Z] End Time: [2018-05-17T02:31:01.276Z] Location: [35.7090259,139.73199249999993] 
Summary: Network ID: 2 Hosts: [4.251.194.212, 74.85.79.113] Total: 2 Start Time: [2018-05-17T02:31:07.280Z] End Time: [2018-05-17T02:31:15.284Z] Location: [40.7127837,-74.0059413] 
Summary: Network ID: 1 Hosts: [49.106.227.186] Total: 1 Start Time: [2018-05-17T02:31:21.288Z] End Time: [2018-05-17T02:31:21.288Z] Location: [37.3860517,-122.0838511] 
Summary: Network ID: 2 Hosts: [84.97.146.31] Total: 1 Start Time: [2018-05-17T02:31:49.307Z] End Time: [2018-05-17T02:31:49.307Z] Location: [40.7127837,-74.0059413] 
Summary: Network ID: 1 Hosts: [126.34.159.25, 47.5.114.210] Total: 2 Start Time: [2018-05-17T02:32:01.313Z] End Time: [2018-05-17T02:32:29.369Z] Location: [37.3860517,-122.0838511] 

- set default scope explicitly to work better in Nautilus environment
- updated help text of anomaly detector
- set stream scale policy to 1 for event time-based apps
- fixed program output of Java-based turbine processor

Signed-off-by: Eron Wright <eronwright@gmail.com>
@EronWright
Copy link
Contributor Author

@vijikarthi I updated each sample to use the following approach to creating the PravegaConfig, and we should do the same in n-samples.

        ParameterTool params = ParameterTool.fromArgs(args);
        PravegaConfig pravegaConfig = PravegaConfig
                .fromParams(params)
                .withDefaultScope("examples");

Meanwhile, the default stream name should be unqualified. This approach will work best when a default scope (PRAVEGA_SCOPE) is in play.

Copy link
Contributor

@vijikarthi vijikarthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@EronWright EronWright merged commit 3ebbec4 into develop May 17, 2018
@EronWright EronWright deleted the issue-76-builder-api branch May 17, 2018 17:45
fpj pushed a commit that referenced this pull request Jun 23, 2018
- rework samples to use the new builder API of the Flink connector
- use `PravegaConfig` to establish a Pravega context (connection info, default scope)
- set stream scale policy to 1 for event time-based apps
- updated help text of anomaly app

Signed-off-by: Eron Wright <eronwright@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants