[FLINK-4450]update storm verion to 1.0.0 in flink-storm and flink-storm-examples #2439

liuyuzhong · 2016-08-30T06:32:00Z

This pull is for update storm version to 1.0.0 https://issues.apache.org/jira/browse/FLINK-4450

Because Apache Storm version after 1.0.0, the pacakge name was changed from "backtype." to "org.apache.", and this change will work for all the storm which version higher than 1.0.0

build successfully and run successfully in my test environment.

liuyuzhong · 2016-08-31T06:56:18Z

mvn clean verify : ok

test flink-storm-examples: ok
bin/flink run WordCount-StormTopology.jar
bin/flink run WordCount-BoltTokenizer.jar
bin/flink run WordCount-SpoutSource.jar

liuyuzhong · 2016-08-31T09:10:06Z

Jenkins run error:

Running org.apache.flink.test.checkpointing.PartitionedStateCheckpointingITCase
SUREFIRE-859: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000d8b00000, 100663296, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 100663296 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/jenkins/jenkins-slave/workspace/flink-github-ci/flink-tests/target/hs_err_pid9818.log
Running org.apache.flink.test.checkpointing.WindowCheckpointingITCase
ERROR: Maven JVM terminated unexpectedly with exit code 137
Putting comment on the pull request
Finished: FAILURE

But I run it successfully in my environment, does it a Jenkins memory problem? Rerun it in Jenkins?

zentol · 2016-08-31T12:09:29Z

...ontrib/flink-storm/src/test/java/org/apache/flink/storm/wrappers/WrapperSetupHelperTest.java

@@ -186,15 +177,15 @@ public void testCreateTopologyContext() {
 				.shuffleGrouping("bolt2", TestDummyBolt.groupingStreamId)
 				.shuffleGrouping("bolt2", TestDummyBolt.shuffleStreamId);

-		LocalCluster cluster = new LocalCluster();


Why `did you modify this code?

It can't work, so delete it first.

Can you elaborate a bit why this cannot work any more with Storm 1.0?

StephanEwen · 2016-08-31T16:29:06Z

There is an interesting question about how to proceed with that.

If we merge this, the compatibility layer will only support Storm 1.0 programs, which are incompatible with prior versions. I would assume a lot of people have Storm jobs with prior versions.

In some sense we would need two compatibility projects, one for Storm pre-1.0 and one for Storm post-1.0.

liuyuzhong · 2016-09-01T01:47:03Z

@StephanEwen
If we merge this, flink-storm will compatible with all the storm which version higher than 1.0.0

And we can import "storm-rename-hack" to compatible with previous storm version. It can change class package name previous storm version jar.

see the detail: https://github.com/apache/storm/tree/master/storm-rename-hack

comments update:

Sorry, It can't work with previous storm version. I will add two new module "flink-storm-post-1.0" "flink-storm-examples-post-1.0" in flink-contrib. What do you think?

liuyuzhong · 2016-09-01T05:03:24Z

close it, and will open new pull request which add new module "flink-apache-storm" "flink-apache-storm-examples" later.

flink-apache-storm: campatible for storm version higher than 1.0.0

liuyuzhong · 2016-09-28T01:30:37Z

Jenkins run successful, but CI fail.

liuyuzhong · 2016-10-10T01:37:31Z

Jenkins run error
1.build success

[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:18 h
[INFO] Finished at: 2016-10-09T12:42:31+00:00
[INFO] Final Memory: 210M/1154M

2.finish fail:

[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/flink-github-ci/flink-contrib/flink-streaming-contrib/target/flink-streaming-contrib_2.10-1.2-SNAPSHOT-javadoc.jar to org.apache.flink/flink-streaming-contrib_2.10/1.2-SNAPSHOT/flink-streaming-contrib_2.10-1.2-SNAPSHOT-javadoc.jar
channel stopped
Putting comment on the pull request
Finished: UNSTABLE

StephanEwen

I think this looks very good. The last question I have before merging is about the dependencies. The flink-storm/pom.xml originally excluded all the Clojure Web Frameworks. That is no longer the case. Did storm 1.0 completely drop these Frameworks (migrate to Java)?

liuyuzhong · 2016-10-12T11:20:08Z

@StephanEwen Actually, I'm not very clear about Clojure Web FrameWork, but it's web framework are using "ring.ring-core".
import: https://github.com/apache/storm/blob/1.x-branch/pom.xml#L703
code detail: https://github.com/apache/storm/blob/1.x-branch/storm-core/src/clj/org/apache/storm/ui/helpers.clj#L36

StephanEwen · 2016-10-12T13:51:33Z

Okay, then we should re-add all the dependency exclusions that were removed from the flink-storm/pom.xml.

StephanEwen · 2016-10-24T16:11:38Z

I would like to merge this. But I think we need to change back the dependency exclusions in flink-storm/pom.xml to as they are in the current master.

Can you do that? Then I will merge this...

liuyuzhong · 2016-10-25T14:03:26Z

Sorry.forget to check it. I will do it tomorrow.

liuyuzhong · 2016-10-28T12:06:14Z

@StephanEwen Added dependency exclusions in flink-storm/pom.xml. Help me to review it please.

StephanEwen · 2016-10-31T14:17:27Z

I think this is getting into good shape, we can almost merge it.
Two things remaining:

Can you rebase/squash all commits into one commit?
There is a complilation error, probably a missing dependency, see below.

Hope that when those are addresses, we can merge this.

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/build/apache/flink/flink-contrib/flink-storm/src/main/java/org/apache/flink/storm/api/FlinkSubmitter.java:[30,23] package org.json.simple does not exist
[INFO] 1 error
[INFO] -------------------------------------------------------------

liuyuzhong · 2016-11-08T12:29:28Z

Fixed complie error.

liuyuzhong · 2016-11-24T14:01:02Z

Complie error is fixed. But how to squash all commits into one commit?

StephanEwen · 2016-11-24T14:04:12Z

Thanks!

You can squash commits by rebasing. Have a look here: http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html

…an shutdown freezed.

…zerUtils This closes apache#2491

…onEnvironment This closes apache#2298.

This closes apache#2515

This closes apache#2694.

…iguration parameters

This closes apache#2709.

…s and bufferPool usage as a metrics This closes apache#2693

This closes apache#2418.

This closes apache#2728.

This closes apache#2718

…nly fails terminally if no more restarts are possible If in state RESTARTING a failure occurs, then a new restart attempt is started. Only if the restart strategy no longer allows further restarts or if the thrown exception is of type SuppressRestartsException a job can go from RESTARTING into FAILED. This closes apache#2710

After emitting a record via the RecordWriter, we eagerly requested a new buffer for the next emit on that channel (although it's not clear that we will immediately need it). With this change, we request that buffer lazily when an emit call requires it. This closes apache#2716.

…rce leaks This closes apache#2665

…scheduleOrUpdateConsumers call Instead of failing the complete ExecutionGraph, a failing scheduleOrUpdateConsumers call will be reported back to the caller. The caller can then decide what to do. Per default, it will fail the calling task. This closes apache#2700

liuyuzhong · 2016-12-05T13:49:53Z

Sorry, something was wrong when I am rebase.

liuyuzhong · 2016-12-22T01:58:21Z

@StephanEwen Sorry, I can't continue with this pull request. I will use another githup account to pull.

liuyuzhong · 2016-12-22T05:14:36Z

@StephanEwen I use new githup account make a new pull request #3037, help me to review it please.Thanks.

greghogan · 2017-01-17T21:03:33Z

Please close this since there is a newer PR.

liuyuzhong · 2017-01-18T01:57:59Z

#3037 has merged. done.

liuyuzhong changed the title ~~update storm verion to 1.0.0 in flink-storm and flink-storm-examples~~ [FLINK-4450]update storm verion to 1.0.0 in flink-storm and flink-storm-examples Aug 30, 2016

liuyuzhong closed this Aug 31, 2016

liuyuzhong reopened this Aug 31, 2016

zentol reviewed Aug 31, 2016
View reviewed changes

liuyuzhong closed this Sep 1, 2016

liuyuzhong mentioned this pull request Sep 7, 2016

[Flink-4450] add a new module "flink-apache-storm" to support new storm verion program (higher than 1.0.0) #2452

Closed

liuyuzhong reopened this Sep 20, 2016

StephanEwen reviewed Oct 10, 2016

View reviewed changes

StephanEwen and others added 5 commits December 5, 2016 21:03

[FLINK-4625] [core] Add a safety net to forcibly terminate JVM is cle…

ab1b7eb

…an shutdown freezed.

remove unnecessary link break

f309f0f

[FLINK-4610] [core] Replace keySet/getValue with entrySet in UdfAnaly…

55ba1b7

…zerUtils This closes apache#2491

[FLINK-4247] [table] CsvTableSource.getDataSet() expects Java Executi…

b96a852

…onEnvironment This closes apache#2298.

[FLINK-4638] [core] Fix exception message for MemorySegment

60b0736

This closes apache#2515

zentol and others added 20 commits December 5, 2016 21:04

[FLINK-4925] [metrics] Integrate meters into IOMetricGroups

07c9335

This closes apache#2694.

[hotfix] [docs] Add documentation for TaskManager's registration conf…

187b7a8

…iguration parameters

[FLINK-4631] Prevent NPE in OneInputStreamTask

c3eb692

This closes apache#2709.

[FLINK-4544[ Refactor JM/TM metrics

5bfff90

#FLINK-4450 add exclusions in flink-storm/pom.xml

40447fa

#FLINK-4450 add exclusions in flink-storm/pom.xml

72b8759

[FLINK-4923] [metrics] Expose Task's input/output buffer queue length…

dedc067

…s and bufferPool usage as a metrics This closes apache#2693

[FLINK-4245] JMXReporter exposes all defined variables

b215249

This closes apache#2418.

[hotfix] [metrics] Make JmxMeter class static

77b6812

[hotfix] Improved test stability of RescalingITCase

b427c81

This closes apache#2728.

[FLINK-4972] Fix CoordinatorShutdownTest

5b99d73

[FLINK-4733] Reuse operator IO metrics for task

f4bbf2b

[FLINK-4733] Port TaskManager metrics

2cb8d0c

[FLINK-4733] Port Task IO metrics

d880e40

[hotfix] Fixes the TimestampedInputSplit.EOS comparison.

c5885e1

This closes apache#2718

[FLINK-4659] [core] Closed the jassConfStream object to prevent resou…

e4474dd

…rce leaks This closes apache#2665

FLINK-4450 add json-simple dependency

d4df2c1

#FLINK-4450 reset version

f509120

liuyuzhong force-pushed the master branch from e722275 to f509120 Compare December 22, 2016 01:54

liuyuzhong7 mentioned this pull request Dec 22, 2016

Flink-4450 update storm version to 1.0 #3037

Closed

liuyuzhong closed this Jan 18, 2017

rmetzger added the component=flink-contrib label Mar 14, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-4450]update storm verion to 1.0.0 in flink-storm and flink-storm-examples #2439

[FLINK-4450]update storm verion to 1.0.0 in flink-storm and flink-storm-examples #2439

liuyuzhong commented Aug 30, 2016 •

edited

Loading

liuyuzhong commented Aug 31, 2016

liuyuzhong commented Aug 31, 2016

zentol Aug 31, 2016

liuyuzhong Sep 26, 2016

StephanEwen Oct 10, 2016

StephanEwen commented Aug 31, 2016

liuyuzhong commented Sep 1, 2016 •

edited

Loading

liuyuzhong commented Sep 1, 2016 •

edited

Loading

liuyuzhong commented Sep 28, 2016

liuyuzhong commented Oct 10, 2016

StephanEwen left a comment

liuyuzhong commented Oct 12, 2016

StephanEwen commented Oct 12, 2016

StephanEwen commented Oct 24, 2016

liuyuzhong commented Oct 25, 2016

liuyuzhong commented Oct 28, 2016 •

edited

Loading

StephanEwen commented Oct 31, 2016

liuyuzhong commented Nov 8, 2016

liuyuzhong commented Nov 24, 2016

StephanEwen commented Nov 24, 2016

liuyuzhong commented Dec 5, 2016

liuyuzhong commented Dec 22, 2016

liuyuzhong commented Dec 22, 2016

greghogan commented Jan 17, 2017

liuyuzhong commented Jan 18, 2017

[FLINK-4450]update storm verion to 1.0.0 in flink-storm and flink-storm-examples #2439

[FLINK-4450]update storm verion to 1.0.0 in flink-storm and flink-storm-examples #2439

Conversation

liuyuzhong commented Aug 30, 2016 • edited Loading

liuyuzhong commented Aug 31, 2016

liuyuzhong commented Aug 31, 2016

zentol Aug 31, 2016

Choose a reason for hiding this comment

liuyuzhong Sep 26, 2016

Choose a reason for hiding this comment

StephanEwen Oct 10, 2016

Choose a reason for hiding this comment

StephanEwen commented Aug 31, 2016

liuyuzhong commented Sep 1, 2016 • edited Loading

liuyuzhong commented Sep 1, 2016 • edited Loading

liuyuzhong commented Sep 28, 2016

liuyuzhong commented Oct 10, 2016

StephanEwen left a comment

Choose a reason for hiding this comment

liuyuzhong commented Oct 12, 2016

StephanEwen commented Oct 12, 2016

StephanEwen commented Oct 24, 2016

liuyuzhong commented Oct 25, 2016

liuyuzhong commented Oct 28, 2016 • edited Loading

StephanEwen commented Oct 31, 2016

liuyuzhong commented Nov 8, 2016

liuyuzhong commented Nov 24, 2016

StephanEwen commented Nov 24, 2016

liuyuzhong commented Dec 5, 2016

liuyuzhong commented Dec 22, 2016

liuyuzhong commented Dec 22, 2016

greghogan commented Jan 17, 2017

liuyuzhong commented Jan 18, 2017

liuyuzhong commented Aug 30, 2016 •

edited

Loading

liuyuzhong commented Sep 1, 2016 •

edited

Loading

liuyuzhong commented Sep 1, 2016 •

edited

Loading

liuyuzhong commented Oct 28, 2016 •

edited

Loading