Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-2305] Add documenation about Storm compatibility layer #884

Closed
wants to merge 1 commit into from
Closed

[FLINK-2305] Add documenation about Storm compatibility layer #884

wants to merge 1 commit into from

Conversation

mjsax
Copy link
Member

@mjsax mjsax commented Jul 2, 2015

additional improvements

  • added RawOutputFormatter
  • bug fix in SimpleOutputFormatter
  • enable default output formatter for StormBoltFileSink

@mjsax mjsax mentioned this pull request Jul 3, 2015
Closed
@mjsax
Copy link
Member Author

mjsax commented Jul 3, 2015

FYI: this PR is related to #878 -> if #878 is merged before, this PR needs an update.

rmetzger
rmetzger reviewed Jul 8, 2015
View reviewed changes
docs/apis/storm_compatibility.md

builder.setSpout("source", new StormFileSpout(inputFilePath));
builder.setBolt("tokenizer", new StormBoltTokenizer()).shuffleGrouping("source");
builder.setBolt("counter", new StormBoltCounter()).fieldsGroupign("tokenizer", new Fields("word"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in fieldsGroupign

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed. :)

@rmetzger
Copy link
Contributor

rmetzger commented Jul 8, 2015

+1 to merge!

[FLINK-2305] Add documenation about Storm compatibility layer
1830778 
additional improvements
- added RawOutputFormatter
- bug fix in SimpleOutputFormatter
- enable default output formatter for StormBoltFileSink
@mjsax mjsax force-pushed the flink-2305-storm-documentation branch from f7dbd7d to 1830778 Compare July 8, 2015 16:12
@StephanEwen
Copy link
Contributor

I just tested the storm compatibility code, and stumbled across the following issues in the logs:

  1. Multiple SLF4J bindings. If Storm brings its own one, we may need to define an exclusion in the storm dependencies.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/cicero/.m2/repository/ch/qos/logback/logback-classic/1.0.13/logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/cicero/.m2/repository/org/slf4j/slf4j-log4j12/1.7.7/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  1. Improper Task life cycle. The logs below indicate that the JobManager and TaskManager are in some way killed/removed before the tasks are properly removed and before the client learns about the JobStatus.

I am a bit puzzled how you actually manage to provoke this ;-)

org.apache.flink.runtime.client.JobExecutionException: Communication with JobManager failed: Lost connection to JobManager akka://flink/user/jobmanager
    at org.apache.flink.runtime.client.JobClient.submitJobAndWait(JobClient.java:169)
    at org.apache.flink.runtime.minicluster.FlinkMiniCluster.submitJobAndWait(FlinkMiniCluster.scala:205)
    at org.apache.flink.runtime.minicluster.FlinkMiniCluster.submitJobAndWait(FlinkMiniCluster.scala:195)
    at org.apache.flink.streaming.util.TestStreamEnvironment$1.run(TestStreamEnvironment.java:116)
Caused by: java.lang.Exception: Lost connection to JobManager akka://flink/user/jobmanager
    at org.apache.flink.runtime.client.JobClientActor.onReceive(JobClientActor.java:131)
    at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
    at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
    at akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
    at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
    at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
    at akka.actor.ActorCell.invoke(ActorCell.scala:486)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
    at akka.dispatch.Mailbox.run(Mailbox.scala:221)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.294 sec - in org.apache.flink.stormcompatibility.exclamation.StormExclamationLocalITCase
21:40:16.300 [exclamation1 (1/3)] ERROR org.apache.flink.runtime.taskmanager.Task  - FATAL - exception in task resource cleanup
java.lang.IllegalStateException: Memory manager has been shut down.
    at org.apache.flink.runtime.memorymanager.DefaultMemoryManager.releaseAll(DefaultMemoryManager.java:396) ~[flink-runtime-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:653) ~[flink-runtime-0.10-SNAPSHOT.jar:0.10-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]

@mjsax
Copy link
Member Author

mjsax commented Jul 8, 2015

  1. Agreed. Should I add this to this PR or open a JIRA and a new PR?
  2. This is a know issues that will be resolved with the new STOP signal.

@StephanEwen
Copy link
Contributor

Let's make it a separate issue.

If there is a followup on these two issues, we can merge this, in my opinion.

@mjsax
Copy link
Member Author

mjsax commented Jul 9, 2015

I just created two JIRAs:

This PR should be ready to get merged.

@StephanEwen
Copy link
Contributor

+1 good to go, with the followup issues being addresses in separate issues.

Merging this...

@StephanEwen
Copy link
Contributor

Ah, just saw it is already merged. Github is out of sync...

@mjsax
Copy link
Member Author

mjsax commented Jul 13, 2015

@gyfora merged this already! PR is not closed because Github is behind afs-git!

@asfgit asfgit closed this in ea4f339 Jul 13, 2015
@mjsax mjsax deleted the flink-2305-storm-documentation branch July 13, 2015 17:23
mxm pushed a commit to mxm/flink that referenced this pull request Jul 14, 2015
@mxm
[FLINK-2305] Add documenation about Storm compatibility layer
4841e19 
additional improvements
- added RawOutputFormatter
- bug fix in SimpleOutputFormatter
- enable default output formatter for StormBoltFileSink

Closes apache#884
shghatge pushed a commit to shghatge/flink that referenced this pull request Aug 8, 2015
@shghatge
[FLINK-2305] Add documenation about Storm compatibility layer
a189cfc 
additional improvements
- added RawOutputFormatter
- bug fix in SimpleOutputFormatter
- enable default output formatter for StormBoltFileSink

Closes apache#884
nikste pushed a commit to nikste/flink that referenced this pull request Sep 29, 2015
@gyfora
[FLINK-2305] Add documenation about Storm compatibility layer
b9b26fd 
additional improvements
- added RawOutputFormatter
- bug fix in SimpleOutputFormatter
- enable default output formatter for StormBoltFileSink

Closes apache#884
nltran pushed a commit to nltran/flink that referenced this pull request Jan 8, 2016
[FLINK-2305] Add documenation about Storm compatibility layer
8da9bee 
additional improvements
- added RawOutputFormatter
- bug fix in SimpleOutputFormatter
- enable default output formatter for StormBoltFileSink

Closes apache#884
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
component=Documentation
Projects
None yet
Milestone
No milestone
3 participants
@mjsax @rmetzger @StephanEwen