Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-2889] Create Hadoop config objects consistently. #1843

Closed
wants to merge 11 commits into from

Commits on Aug 7, 2014

  1. Replace explicit Configuration instantiation with SparkHadoopUtil.

    This is the basic grunt work; code doesn't fully compile yet, since
    I'll do some of the more questionable changes in separate commits.
    Marcelo Vanzin committed Aug 7, 2014
    Configuration menu
    Copy the full SHA
    1e7003f View commit details
    Browse the repository at this point in the history
  2. Update Utils API to take a Configuration argument.

    Instead of using "new Configuration()" where a configuration is
    needed, let the caller provide a context-appropriate config
    object.
    Marcelo Vanzin committed Aug 7, 2014
    Configuration menu
    Copy the full SHA
    b8ab173 View commit details
    Browse the repository at this point in the history
  3. Initialize config in SparkHadoopUtil.

    This is sort of hackish, since it doesn't account for any customization
    someone might make to SparkConf before they actually start executing spark
    code. Instead, this will only consider options available in the
    system properties when creating the hadoop conf.
    Marcelo Vanzin committed Aug 7, 2014
    Configuration menu
    Copy the full SHA
    f16cadd View commit details
    Browse the repository at this point in the history
  4. Compilation fix.

    Marcelo Vanzin committed Aug 7, 2014
    Configuration menu
    Copy the full SHA
    3f26760 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2014

  1. Merge branch 'master' into SPARK-2889

    Conflicts:
    	core/src/main/scala/org/apache/spark/util/FileLogger.scala
    	yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
    Marcelo Vanzin committed Aug 20, 2014
    Configuration menu
    Copy the full SHA
    0ac3fdf View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2014

  1. Merge branch 'master' into SPARK-2889

    Conflicts:
    	core/src/main/scala/org/apache/spark/scheduler/cluster/SimrSchedulerBackend.scala
    Marcelo Vanzin committed Aug 26, 2014
    Configuration menu
    Copy the full SHA
    fc45067 View commit details
    Browse the repository at this point in the history
  2. Restore old method for backwards compat.

    Marcelo Vanzin committed Aug 26, 2014
    Configuration menu
    Copy the full SHA
    3d345cb View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2014

  1. Add DeveloperApi annotation.

    Marcelo Vanzin committed Aug 27, 2014
    Configuration menu
    Copy the full SHA
    53f9506 View commit details
    Browse the repository at this point in the history
  2. Add test to ensure that overriding Yarn configs works.

    Marcelo Vanzin committed Aug 27, 2014
    Configuration menu
    Copy the full SHA
    51e71cf View commit details
    Browse the repository at this point in the history
  3. Merge branch 'master' into SPARK-2889

    Conflicts:
    	yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
    	yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala
    	yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientClusterScheduler.scala
    	yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala
    	yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
    	yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala
    Marcelo Vanzin committed Aug 27, 2014
    Configuration menu
    Copy the full SHA
    f179013 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2014

  1. Merge branch 'master' into SPARK-2889

    Conflicts:
    	core/src/test/scala/org/apache/spark/scheduler/ReplayListenerSuite.scala
    Marcelo Vanzin committed Aug 29, 2014
    Configuration menu
    Copy the full SHA
    52daf35 View commit details
    Browse the repository at this point in the history