Permalink
Browse files

So jvm job reuse should be finite, I've learned to my chagrin

  • Loading branch information...
1 parent 613f2f7 commit 2523670c89e0559cc4f5d421b9fc0551c441f06b Philip (flip) Kromer committed Jan 14, 2014
Showing with 10 additions and 2 deletions.
  1. +1 −1 22-hadoop_tuning.asciidoc
  2. +2 −1 23-hadoop_tuning-brave_and_foolish.asciidoc
  3. +7 −0 25c-references.asciidoc
@@ -117,7 +117,7 @@ The second job ran its 10,000 Map tasks through a purposefully restricted 50 Map
(TODO: show how to find out if one node is way slower)
-Even in this trivial case, there is more variance in launch and runtimes than you might first suspect (if you don't, you definitely will in the next -- but for continuity, we will discuss it here). If that splay -- the delay between the bulk of jobs finishing and the final job finishing -- is larger than the runtime of a typical task, however, it may indicate a problem, but as long as it is only a few seconds, don’t sweat it. If you are interested in a minor but worth-it tweak, adjust the `mapred.job.reuse.jvm.num.tasks` setting to ‘-1’, which causes each Mapper to use the same child process JVM for its attempts, eliminating the brief but noticeable JVM startup time. If you are writing your own native Java code, you might know a reason to go with the default (disabled) setting but it is harmless for any well-behaved program.
+Even in this trivial case, there is more variance in launch and runtimes than you might first suspect (if you don't, you definitely will in the next -- but for continuity, we will discuss it here). If that splay -- the delay between the bulk of jobs finishing and the final job finishing -- is larger than the runtime of a typical task, however, it may indicate a problem, but as long as it is only a few seconds, don’t sweat it. If you are interested in a minor but worth-it tweak, adjust the `mapred.job.reuse.jvm.num.tasks` setting to ‘10’. This causes each Mapper to use the same child process JVM for multiple attempts, minimizing the brief but noticeable JVM startup time's impact. If you are writing your own native Java code, you might know a reason to force no reuse (the default), but it is generally harmless for any well-behaved program.
On the Job screen, you should see that the total runtime for the job was about 200 times slower for the second job than the first and not much more than 200 times the typical task’s runtime; if not, you may be putting pressure on the Job Tracker. Rerun your job and watch the Job Tracker’s heap size; you would like the Job Tracker heap to spend most of its life below, say 50-percent, so if you see it making any significant excursions toward 100-percent, that would unnecessarily impede cluster performance. The 1 GB out-of-the-box setting is fairly small; for production use we recommend at least 3 GB of heap on a dedicated machine with at least 7 GB total RAM.
@@ -43,7 +43,7 @@ Here's a plausible configuration for a 16-GB physical machine with 8 cores:
* `mapred.reduce.parallel.copies`: default `X`, recommended to be in the range of `sqrt(Nw*Nm)` to `Nw*Nm/2` You should see the shuffle/copy phase of your reduce tasks speed up.
-* `mapred.job.reuse.jvm.num.tasks` default `1`, recommended `-1`. If a job requires a fresh JVM for each process, you can override that in its jobconf.
+* `mapred.job.reuse.jvm.num.tasks` default `1`, recommended `10`. If a job requires a fresh JVM for each process, you can override that in its jobconf. Going to `-1` (reuse unlimited times) can fill up the dist if your input format uses "delete on exit" temporary files (as for example the S3 filesystem does), with little additional speedup.
* You never want Java to be doing stop-the-world garbage collection, but for large JVM heap sizes (above 4GB) they can become especially dangerous. If a full garbage collect takes too long, sockets can time out, causing loads to increase, causing garbage collects to happen, causing... trouble, as you can guess.
@@ -110,3 +110,4 @@ Here's a plausible configuration for a 16-GB physical machine with 8 cores:
* Bump `mapreduce.job.counters.limit` -- it's not configurable per-job.
+(From http://blog.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/ -- 512M block size fairly reasonable)
View
@@ -10,6 +10,13 @@
* http://hadoop.apache.org/docs/mapreduce/current/streaming.html[Hadoop Streaming FAQ]
* http://hadoop.apache.org/docs/r0.20.2/mapred-default.html[Hadoop Configuration defaults -- mapred]
+
+**Unreasonable Effectiveness of Data**
+
+* https://www.facebook.com/video/video.php?v=644326502463[Peter Norvig\'s Facebook Tech Talk]
+* http://www.youtube.com/watch?v=yvDCzhbjYWs[Later version of that talk at ??] -- I like the
+* http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/35179.pdf["On the Unreasonable effectiveness of data"]
+
**Source material**
* http://en.wikipedia.org/wiki/Lexington,_Texas[Wikipedia article on Lexington, Texas] (CC-BY-SA)

0 comments on commit 2523670

Please sign in to comment.