Skip to content

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also compare across forks.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also compare across forks.
...
Commits on Jan 29, 2013
Imran Rashid track task completion in DAGScheduler, and send a stageCompleted even…
…t with taskInfo to SparkListeners
b148414
Imran Rashid simple util to summarize distributions b88daee
Imran Rashid can get task runtime summary from task info 38b83bc
Imran Rashid expose stageInfo in SparkContext 01d77f3
Imran Rashid convenient name available in StageInfo cec9c76
Commits on Feb 05, 2013
Imran Rashid Merge branch 'master' into stageInfo
Conflicts:
	core/src/main/scala/spark/scheduler/DAGScheduler.scala
	core/src/main/scala/spark/scheduler/local/LocalScheduler.scala
b430d23
Imran Rashid track total bytes written by ShuffleMapTasks 843084d
Imran Rashid add TimedIterator 1ad77c4
Imran Rashid Shuffle Fetchers use a timed iterator 9df7e2a
Imran Rashid task context keeps a handle on Task -- giant hack, temporary for trac…
…king shuffle times & amount
295b534
Imran Rashid cogrouped RDD stores the amount of time taken to read shuffle data in…
… each task
e319ac7
Imran Rashid BlockManager.getMultiple returns a custom iterator, to enable trackin…
…g of shuffle performance
b29f9cc
Imran Rashid track remoteFetchTime 696e4b2
Imran Rashid add as many fetch requests as we can, subject to maxBytesInFlight 1704b12
Commits on Feb 06, 2013
Imran Rashid setup plumbing to get task metrics; lots of unfinished parts, but bas…
…ic flow in place
379564c
Commits on Feb 09, 2013
Imran Rashid general fixes to Distribution, plus some tests 04e828f
@stephenh stephenh Use stubs instead of mocks for DAGSchedulerSuite. 921be76
Commits on Feb 10, 2013
Imran Rashid use TaskMetrics to gather all stats; lots of plumbing to get it all t…
…he way back to driver
b7d9e24
Imran Rashid SparkContext.addSparkListener; "std" listener in StatsReportListener 383af59
Commits on Feb 11, 2013
Imran Rashid cleanup a bunch of imports d9461b1
Imran Rashid undo chnage to onCompleteCallbacks e9f53ec
Commits on Feb 15, 2013
Imran Rashid Merge branch 'master' into stageInfo
Conflicts:
	core/src/main/scala/spark/rdd/CoGroupedRDD.scala
	core/src/main/scala/spark/storage/BlockManager.scala
bffee92
Commits on Feb 21, 2013
Imran Rashid TaskContext does not hold a reference to Task; instead, it has a shar…
…ed instance of TaskMetrics with Task
baab23a
Imran Rashid fully revert change to addOnCompleteCallback -- missed this in e9f53ec 69f9a70
Imran Rashid Merge branch 'master' into stageInfo
Conflicts:
	core/src/main/scala/spark/SparkContext.scala
	core/src/main/scala/spark/storage/BlockManager.scala
ff127cf
Imran Rashid add timing around parts of executor & track result size f2fcabf
Imran Rashid add task result size; better formatting for time interval distributio…
…ns; cleanup distribution formatting
176cb20
Imran Rashid add runtime breakdowns 6f62a57
Imran Rashid taskInfo tracks if a task is run on a preferred host d0bfac3
Imran Rashid get rid of a bunch of boilerplate; more formatting happens in Listene…
…r, not StageInfo
7960927
Imran Rashid store taskInfo & metrics together in a tuple 394d3ac
Imran Rashid add some docs & some cleanup 796e934
Imran Rashid sparkListeners should be a val 81bd07d
Commits on Feb 22, 2013
Imran Rashid add cleanup iterator 9230617
Imran Rashid make the ShuffleFetcher responsible for collecting shuffle metrics, w…
…hich gives us metrics for CoGroupedRDD and ShuffledRDD
0f37b43
Commits on Feb 25, 2013
Imran Rashid remove bogus comment 8f17387
Commits on Feb 26, 2013
@stephenh stephenh Merge branch 'master' into nomocks
Conflicts:
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
a4adeb2
@stephenh stephenh Override DAGScheduler.runLocally so we can remove the Thread.sleep. a65aa54
@stephenh stephenh Fix MapOutputTrackerSuite. db957e5
Commits on Feb 27, 2013
@mateiz mateiz Added commented-out Google analytics code for website docs 4f840f4
@mateiz mateiz Change version to 0.7.1-SNAPSHOT for development branch db9b90f
@mosharaf mosharaf Fixed master datastructure updates after removing an application; and…
… a typo.
4ab387b
Commits on Feb 28, 2013
@rxin rxin Fixed SPARK-706: Failures in block manager put leads to read task
hanging.
44134e1
Commits on Mar 01, 2013
@markhamstra markhamstra bump version to 0.7.1-SNAPSHOT in the subproject poms to keep the mav…
…en build building.
8b06b35
@mateiz mateiz Merge pull request #507 from markhamstra/poms271
bump version to 0.7.1-SNAPSHOT in the subproject poms
25c71d3
@markhamstra markhamstra Instead of failing to bind to a fixed, already-in-use port, let the O…
…S choose an available port for TestServer.
b409073
Commits on Mar 03, 2013
@mateiz mateiz Merge pull request #508 from markhamstra/TestServerInUse
Avoid bind failure in InputStreamsSuite
94b3db1
@mateiz mateiz Merge pull request #504 from mosharaf/master
Worker address was getting removed when removing an app.
6bfc7ca
Imran Rashid Merge branch 'master' into stageInfo d36abdb
Commits on Mar 04, 2013
Imran Rashid refactoring of TaskMetrics 8fef5b9
Imran Rashid change CleanupIterator to CompletionIterator f1006b9
Imran Rashid minor cleanup based on feedback in review request 0bd1d00
@mateiz mateiz Merge pull request #462 from squito/stageInfo
Track assorted metrics for each task, report summaries to user at stage completion
6cf4be4
@mateiz mateiz Merge pull request #506 from rxin/spark-706
Fixed SPARK-706: Failures in block manager put leads to read task hanging.
04fb81f
@mateiz mateiz Fix TaskMetrics not being serializable 9f0dc82
Commits on Mar 08, 2013
@patelh patelh Fix reference bug in Kryo serializer, add test, update version 664e5fd
Commits on Mar 09, 2013
@MLnick Added choice of persitance level to Bagel. Also added documentation. 1e981d8
@MLnick Adding test for non-default persistence level d35c5a5
@woggle woggle Prevent DAGSchedulerSuite from corrupting driver.port.
Use the LocalSparkContext abstraction to properly manage clearing
spark.driver.port.
d0216cb
@woggle woggle Notify standalone deploy client of application death.
Usually, this isn't necessary since the application will be removed
as a result of the deploy client disconnecting, but occassionally, the
standalone deploy master removes an application otherwise.

Also mark applications as FAILED instead of FINISHED when they are
killed as a result of their executors failing too many times.
b0983c5
Commits on Mar 10, 2013
@mateiz mateiz Merge remote-tracking branch 'stephenh/nomocks'
Conflicts:
	core/src/main/scala/spark/storage/BlockManagerMaster.scala
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
a59cc60
@mateiz mateiz Merge pull request #515 from woggling/deploy-app-death
Notify standalone deploy client of application death.
557cfd0
@mateiz mateiz Merge pull request #512 from patelh/fix-kryo-serializer
Fix reference bug in Kryo serializer, add test, update version
91a9d09
@mateiz mateiz Update kryo-serializers version in pom.xml to match previous commit d4e29ea
@mateiz mateiz Merge remote-tracking branch 'woggling/dag-sched-driver-port'
Conflicts:
	core/src/test/scala/spark/scheduler/DAGSchedulerSuite.scala
2e1bbc4
Commits on Mar 11, 2013
@MLnick Fix doc style 8dd943f
@mateiz mateiz Merge pull request #513 from MLnick/bagel-caching
Adds choice of persistence level to Bagel.
cbf8f0d