Support loading config from multiple files.

This subsumes two disparate pieces of functionality:

A) PANTS_CONFIG_OVERRIDES.
B) rcfiles.

A) This env variable has been retconned int a --config-overrides
   bootstrap option.

B) Rcfiles used to contain extra command-line flags for various commands, which
   was weird. Since all flags are now options, and so can be set from config,
   the new rcfiles are just regular pants.ini-style config files.

After deploying pants with this commit, existing rcfiles will need to be converted
to pants.ini style. However currently rcfiles are broken anyway, so this isn't
an onerous requirement.

This commit also includes various changes needed to get things to work under
this new config scheme. Some of these are temporary hacks that will go away
when we get rid of Command, or when we get rid of all direct accesses to config.

One prominent change is a modification to config caching. Previously
configs were keyed by the file they were read from. But in practice
code always calls Config.from_cache() with no argument, which
defaults to the regular pants.ini. This leads to dangerous confusion
when config can be read from other files. So now there's just a single
global cached config, which must be set explicitly. Again, this will
go away once we convert all direct config accesses to options accesses,
and plumb options through to all the necessary places.

Testing Done:
CI passes.

Reviewed at https://rbcommons.com/s/twitter/r/1442/

src/docs/invoking.md

@@ -55,60 +55,3 @@ passthrough args to tools called by all of `test`, not just `test.pytest`.
 This gets tricky. We happen to know we can pass `-k list` here, that `pytest` accepts passthrough
 args, and that this `test` invocation upon `tests/python/...` won't invoke any other tools
 (assuming nobody hid a `junit` test in the `python` directory).
-
-rc files
---------
-
-(As of November 2014, rc file support was changing. Expect this
-info to be obsolete soon:)
-
-If there's a command line flag that you always (or nearly always) use,
-you might set up a configuration file to ease this. A typical Pants
-installation looks for machine-specific settings in `/etc/pantsrc` and
-personal settings in `~/.pants.rc`, with personal settings overriding
-machine-specific settings.
-
-For example, suppose that every time you invoke Pants to compile Java
-code, you pass flags
-`--compile-javac-args=-source --compile-javac-args=7 --compile-javac-args=-target --compile-javac-args=7`.
-Instead of passing them on the command line each time, you could set up
-a `~/.pants.rc` file:
-
-    :::ini
-    [javac]
-    options:
-      --compile-javac-args=-source --compile-javac-args=7
-      --compile-javac-args=-target --compile-javac-args=7
-
-With this configuration, Pants will have these flags on by default.
-
-`--compile-javac-*` flags go in the `[javac]` section; generally,
-`--compile`-*foo*-\* flags go in the `[foo]` section. `--test-junit-*`
-flags go in the `[junit]` section; generally, `--test`-*bar*-\* flags go
-in the `[bar]` section. `--idea-*` flags go in the `[idea]` section.
-
-If you know the Pants internals well enough to know the name of a `Task`
-class, you can use that class' name as a category to set command-line
-options affecting it:
-
-    :::ini
-    [pants.tasks.nailgun_task.NailgunTask]
-    # Don't spawn compilation daemons on this shared build server
-    options: --no-ng-daemons
-
-Although `/etc/pantsrc` and `~/.pants.rc` are the typical places for
-this configuration, you can check pants.ini \<setup-pants-ini\> to find
-out what your source tree uses.
-
-    :::ini
-    # excerpt from pants.ini
-    [DEFAULT]
-    # Look for these rcfiles - they need not exist on the system
-    rcfiles: ['/etc/pantsrc', '~/.pants.CUSTOM.rc'] # different .rc name!
-
-In this list, later files override earlier ones.
-
-These files are formatted as [Python config
-files](http://docs.python.org/install/index.html#inst-config-syntax),
-parsed by
-[ConfigParser](http://docs.python.org/library/configparser.html).

src/python/pants/backend/jvm/tasks/scalastyle.py

@@ -52,7 +52,6 @@ def register_options(cls, register):
 
   def __init__(self, *args, **kwargs):
     super(Scalastyle, self).__init__(*args, **kwargs)
-
     self._initialize_config()
     self._scalastyle_bootstrap_key = 'scalastyle'
     self.register_jvm_tool(self._scalastyle_bootstrap_key, ['//:scalastyle'])

src/python/pants/base/BUILD

@@ -140,6 +140,7 @@ python_library(
   sources = ['config.py'],
   dependencies = [
     ':build_environment',
+    'src/python/pants/util:strutil',
   ]
 )
 
@@ -228,16 +229,6 @@ python_library(
   ]
 )
 
-python_library(
-  name = 'rcfile',
-  sources = ['rcfile.py'],
-  dependencies = [
-    ':config',
-    '3rdparty/python/twitter/commons:twitter.common.lang',
-    '3rdparty/python/twitter/commons:twitter.common.log',
-  ]
-)
-
 python_library(
   name = 'revision',
   sources = ['revision.py'],

src/python/pants/base/config.py

@@ -4,16 +4,19 @@
 
 from __future__ import (nested_scopes, generators, division, absolute_import, with_statement,
                         print_function, unicode_literals)
+
 try:
   import ConfigParser
 except ImportError:
   import configparser as ConfigParser
 
 import copy
-import os
 import getpass
+import itertools
+import os
 
 from pants.base.build_environment import get_buildroot
+from pants.util.strutil import is_text_or_binary
 
 
 def reset_default_bootstrap_option_values(defaults, values=None):
@@ -72,32 +75,47 @@ class ConfigError(Exception):
   def reset_default_bootstrap_option_values(cls, values=None):
     reset_default_bootstrap_option_values(cls._defaults, values)
 
-  _cache = {}
+  _cached_config = None
+
+  @classmethod
+  def _munge_configpaths_arg(cls, configpaths):
+    """Converts a string or iterable-of-strings argument into a tuple of strings.
+
+    Result is hashable, so may be used as a cache key.
+    """
+    if is_text_or_binary(configpaths):
+      return (configpaths,)
+    return tuple(configpaths) if configpaths else (os.path.join(get_buildroot(), 'pants.ini'),)
 
   @classmethod
-  def clear_cache(cls):
-    cls._cache = {}
+  def from_cache(cls):
+    if not cls._cached_config:
+      raise cls.ConfigError('No config cached.')
+    return cls._cached_config
 
   @classmethod
-  def from_cache(cls, configpath=None):
-    configpath = configpath or os.path.join(get_buildroot(), 'pants.ini')
-    if configpath not in cls._cache:
-      cls._cache[configpath] = cls.load(configpath)
-    return cls._cache[configpath]
+  def cache(cls, config):
+    cls._cached_config = config
 
   @classmethod
-  def load(cls, configpath=None):
-    """Loads a Config from the given path.
+  def load(cls, configpaths=None):
+    """Loads config from the given paths.
 
      By default this is the path to the pants.ini file in the current build root directory.
+     Callers may specify a single path, or a list of the paths of configs to be chained, with
+     later instances taking precedence over eariler ones.
+
      Any defaults supplied will act as if specified in the loaded config file's DEFAULT section.
      The 'buildroot', invoking 'user' and invoking user's 'homedir' are automatically defaulted.
     """
-    configpath = configpath or os.path.join(get_buildroot(), 'pants.ini')
-    parser = cls.create_parser()
-    with open(configpath) as ini:
-      parser.readfp(ini)
-    return Config(parser)
+    configpaths = cls._munge_configpaths_arg(configpaths)
+    single_file_configs = []
+    for configpath in configpaths:
+      parser = cls.create_parser()
+      with open(configpath, 'r') as ini:
+        parser.readfp(ini)
+      single_file_configs.append(SingleFileConfig(configpath, parser))
+    return ChainedConfig(single_file_configs)
 
   @classmethod
   def create_parser(cls, defaults=None):
@@ -109,33 +127,8 @@ def create_parser(cls, defaults=None):
     standard_defaults = copy.copy(cls._defaults)
     if defaults:
       standard_defaults.update(defaults)
-
     return ConfigParser.SafeConfigParser(standard_defaults)
 
-  def __init__(self, configparser):
-    self.configparser = configparser
-
-    # Overrides
-    #
-    # This feature allows a second configuration file which will override
-    # pants.ini to be specified.  The file is currently specified via an env
-    # variable because the cmd line flags are parsed after config is loaded.
-    #
-    # The main use of the extra file is to have different settings based on
-    # the environment.  For example, the setting used to compile or locations
-    # of caches might be different between a developer's local environment
-    # and the environment used to build and publish artifacts (e.g. Jenkins)
-    #
-    # The files cannot reference each other's values, so make sure each one is
-    # internally consistent
-    self.overrides_path = os.environ.get('PANTS_CONFIG_OVERRIDE')
-    self.overrides_parser = None
-    if self.overrides_path is not None:
-      self.overrides_path = os.path.join(get_buildroot(), self.overrides_path)
-      self.overrides_parser = Config.create_parser()
-      with open(self.overrides_path) as o_ini:
-        self.overrides_parser.readfp(o_ini, filename=self.overrides_path)
-
   def getbool(self, section, option, default=None):
     """Equivalent to calling get with expected type string"""
     return self.get(section, option, type=bool, default=default)
@@ -189,26 +182,10 @@ def get_required(self, section, option, type=str):
       raise Config.ConfigError('Required option %s.%s is not defined.' % (section, option))
     return val
 
-  def has_section(self, section):
-    """Return whether or not this config has the section."""
-    return self.configparser.has_section(section)
-
-  def has_option(self, section, option):
-    if self.overrides_parser and self.overrides_parser.has_option(section, option):
-      return True
-    elif self.configparser.has_option(section, option):
-      return True
-    return False
-
-  def _get_value(self, section, option):
-    if self.overrides_parser and self.overrides_parser.has_option(section, option):
-      return self.overrides_parser.get(section, option)
-    return self.configparser.get(section, option)
-
   def _getinstance(self, section, option, type, default=None):
     if not self.has_option(section, option):
       return default
-    raw_value = self._get_value(section, option)
+    raw_value = self.get_value(section, option)
     if issubclass(type, str):
       return raw_value
 
@@ -223,3 +200,77 @@ def _getinstance(self, section, option, type, default=None):
         type.__name__, section, option, raw_value))
 
     return parsed_value
+
+  # Subclasses must implement.
+
+  def sources(self):
+    """Return the sources of this config as a list of filenames."""
+    raise NotImplementedError()
+
+  def has_section(self, section):
+    """Return whether this config has the section."""
+    raise NotImplementedError()
+
+  def has_option(self, section, option):
+    """Return whether this config specified a value the option."""
+    raise NotImplementedError()
+
+  def get_value(self, section, option):
+    """Return the value of the option in this config, as a string, or None if no value specified."""
+    raise NotImplementedError()
+
+
+class SingleFileConfig(Config):
+  """Config read from a single file."""
+  def __init__(self, configpath, configparser):
+    super(SingleFileConfig, self).__init__()
+    self.configpath = configpath
+    self.configparser = configparser
+
+  def sources(self):
+    return [self.configpath]
+
+  def has_section(self, section):
+    return self.configparser.has_section(section)
+
+  def has_option(self, section, option):
+    return self.configparser.has_option(section, option)
+
+  def get_value(self, section, option):
+    return self.configparser.get(section, option)
+
+
+class ChainedConfig(Config):
+  """Config read from multiple sources."""
+  def __init__(self, configs):
+    """
+    :param configs: A list of Config instances to chain.
+                    Later instances take precedence over earlier ones.
+    """
+    super(ChainedConfig, self).__init__()
+    self.configs = list(reversed(configs))
+
+  def sources(self):
+    return list(itertools.chain.from_iterable(cfg.sources() for cfg in self.configs))
+
+  def has_section(self, section):
+    for cfg in self.configs:
+      if cfg.has_section(section):
+        return True
+    return False
+
+  def has_option(self, section, option):
+    for cfg in self.configs:
+      if cfg.has_option(section, option):
+        return True
+    return False
+
+  def get_value(self, section, option):
+    for cfg in self.configs:
+      try:
+        return cfg.get_value(section, option)
+      except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+        pass
+    if not self.has_section(section):
+      raise ConfigParser.NoSectionError(section)
+    raise ConfigParser.NoOptionError(option, section)