New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAHOUT-1750 - For Review. FlinkOpABt too many operations to implement? #215

Closed
wants to merge 116 commits into
base: flink-binding
from

Conversation

Projects
None yet
6 participants
@andrewpalumbo
Member

andrewpalumbo commented Apr 10, 2016

When (partially) Implementing FlinkOpABt so far, It seems that there may be too many Flink map/reduce/group/etc operations to finish using this method. NOTE: this is unfinished and is not numerically correct.

Currently, when testing am getting kyro stack overflow exceptions, which as I understand are often caused by a long string of operations:

Job execution failed.
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
    at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:716)
    at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662)
    at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.StackOverflowError
    at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:74)
    at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
    at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:57)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
    at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
{...}

Any comments are appreciated.

smarthi and others added some commits Oct 25, 2015

MAHOUT-1775 FileNotFoundException caused by aborting the process of d…
…ownloading Wikipedia dataset, closes apache/mahout# 162
MAHOUT-1775 FileNotFoundException caused by aborting the process of d…
…ownloading Wikipedia dataset, closes apache/mahout #162
MAHOUT-1772: delimiterPattern instance variable of FileDataModel shou…
…ld be marked as a Transient field, this closes #160
Support environment variable MAHOUT_WORK_DIR to allow users to
specify an alternate temporary directory in examples.

smarthi and others added some commits Apr 13, 2016

reverting commits:
commit df1f064
Author: Andrew Palumbo <apalumbo@apache.org>
Date:   Tue Apr 19 20:20:11 2016 -0400

    (nojira): follow dl4j's travis configuration using a 30 second ping to keep long tests alive

commit 48f05d6
Author: Andrew Palumbo <apalumbo@apache.org>
Date:   Tue Apr 19 19:27:23 2016 -0400

    (nojira) fix double install entry in .travis.yml.  also fix unreasonable memory request

commit 3fe0061
Author: Andrew Palumbo <apalumbo@apache.org>
Date:   Tue Apr 19 19:17:21 2016 -0400

    (Nojira) minor fix to readme to trigger travis build.

commit 9fcebb0
Author: Andrew Palumbo <apalumbo@apache.org>
Date:   Tue Apr 19 19:07:45 2016 -0400

    (nojira) .travis.yml add travis_wait for timing out tests, use some settings from jenkins as MAVEN_OPTS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment