Server configuration

John Collins edited this page Apr 14, 2014 · 23 revisions
Clone this wiki locally

up

Server configuration needs to be done at two different levels: The server itself must be informed about the session (or sessions) it is expected to run, and the individual modules must be configured in various ways.

Session configuration

A server can be run from a shell command-line, or it may be run under control of a Tournament Scheduler, and in either case it can be asked to run bootstrap or sim sessions. In version 0.1.0, the server takes a single command-line argument specifying a file, which in turn contains a sequence of very simple session specifications. This is causing problems as we start to think through how the server will interact with the Tournament Scheduler. So here is a proposal for an enriched CLI that should support the needs of both developers and the Tournament Scheduler:

Bootstrap mode

server --boot bootstrap-data [options]

where

  • bootstrap-data is the name (not a URL) of the xml file that will be written with the results of the bootstrap run,
  • options include:
    • --control controller-url gives the url of the Tournament Scheduler api, from which the server can request a configuration and a log-prefix string.
    • --config server-config giving the URL of a properties file that overrides the standard server configuration. If this option is missing and the --control option is given, the server configuration is retrieved from controller-url/server-config.
    • --log-suffix suffix gives the root name for the log files, and defaults to "boot"; two log files are produced: powertac-suffix.trace and powertac-suffix.state. If this option is missing and --control is given, the logfile prefix will be retrieved from controller-url/log-suffix.

Sim mode

server --sim [options]

where options include the --config, --log-suffix, and --control options as in bootstrap mode, as well as

  • --boot-data bootstrap-data gives the URL of the xml file from which a bootstrap record can be read. If this option is missing and the --control option is given, then the URL for the bootstrap record will be controller-url/bootstrap-data. Note: the server will not start if one of these two sources does not produce a valid bootstrap dataset.
  • --jms-url url gives the URL of the jms message broker, which is typically, but not necessarily, instantiated inside the server. The default value is tcp://localhost:61616.
  • --brokers broker,... is a comma-separated list of broker usernames that are expected to log in to the simulation before it starts. If this option is missing and --control is provided, then the broker list will be retrieved from controller-url/broker-list.
  • --log-suffix defaults to "sim" rather than "boot".

Module configuration

The Power TAC server is a collection of configurable modules. The goal of this discussion is to arrive at a clear, consistent set of principles for how, when, and from where various configuration information is introduced to the server.

Here are some goals for an ideal configuration model:

  • Information about what properties and behaviors are configurable should be readily accessible and understandable by all users who might have an interest in creating a non-standard configuration. This includes users who wish to create configurations by editing text files, those who wish to edit configurations through a simple web interface on the server, and those who wish to configure tournament or experimental games to be run at a later time.
  • Since the server is modular, configuration data sources should also be modular.
  • Configurability should not introduce unnecessary duplication. Ideally, the presence of a setter method with appropriate annotation and documentation would be enough. Otherwise, there is always the problem of keeping configurable code, documentation (including descriptions on web pages) and the config files in sync.
  • It should not be necessary to maintain a global list of types that need to be configured. Rather, configurable types (or appropriate proxies) should ask to be configured during initialization. This is a feature of the old Apache Avalon/Excalibur framework, and a primary reason why the Spring configuration scheme is not adequate.
  • Reasonable default values should be provided and documented. Ideally these are simply hard-coded to avoid duplication and extra work.
  • A reasonable set of types for configuration values should be available, including String, integer and floating numbers, lists, and perhaps maps.
  • Constraints on configuration values should be automatically communicated and applied to the extent possible.
  • It should be possible to write expressions for configuration values that include the values of other configuration values as variables.
  • Configuration expressions should be able to create and configure new instances of domain types, such as Gencos.
  • Some configuration information needs to be packaged up and sent to brokers at the beginning of a game. This should also be the responsibility of the configurable types, not some centralized service. This is especially true when runtime parameters are computed from configured values, for example if configuration specified min and max values for some interest rate or charge, but brokers need to know the computed value.

Design approach

There are several possible approaches to this problem.

  • The Excalibur framework includes a "Configurable" interface; at a specific point in the lifecycle of a Configurable component, an inherited method is called with its portion of the overall configuration tree. The keys in the configuration at tied to "role names" in the Excalibur setup.
  • A top-down approach, in which some sort of mapping (possibly hard-coded) is used to map configuration data to components and their properties or setter methods.
  • A bottom-up approach, as used in Magnet and in PowerTAC up to now, in which components invoke methods on a central 'properties' service to extract configuration data.
  • An annotation-based approach, in which configurable elements are identified by annotation, and generic code reads the config data and applies the configurations.

We have chosen an annotation-based scheme. The design includes four elements:

  1. A set of annotation types. @ConfigurableValue labels a setter method or field as a configuration point, and @ConfigurableInstance labels a class as capable of creating multiple configurable instances. If the value is to be published to brokers, then the publish property of the annotation must be set to true.
  2. A service called a Configurator that takes a configuration (specifically, a commons.configuration.Configuration) and a configurable instance or class, and applies the configurations by processing annotations.
  3. A set of configuration files on the classpath (in the src/main/resources directories, or in the jarfiles, in either properties format identified as config/*.properties, or xml format identified as config/properties.xml). Note that Java is not able to reliably find resources using wildcard searches in jarfiles in the classpath root (in other words, filenames at the top level under src/main/resources).
  4. An optional user-supplied configuration file (or URL) that overrides properties set from classpath resources.
  5. A service called ServerPropertiesService that reads the configuration files, merges them into a single Configuration, and waits for entities to request configuration and publish their settings for brokers.

Given the annotations and the ability to apply them, much of the rest can be provided by Apache Commons Configuration and some glue code.

The configuration data is published to brokers in the form of a Properties instance. Here's an example:

    {tariffmarket.tariffMarketService.revocationFee=-133.8693054289303,
     tariffmarket.tariffMarketService.publicationFee=-330.73270535433323, 
     auctioneer.auctionService.sellerSurplusRatio=0.5,
     auctioneer.auctionService.defaultClearingPrice=40.0, 
     auctioneer.auctionService.defaultMargin=0.2, 
     accounting.accountingService.bankInterest=0.061056334512389523, 
     distributionutility.distributionUtilityService.defaultSpotPrice=-50.0, 
     distributionutility.distributionUtilityService.balancingCost=-0.04914701992562123, 
     distributionutility.distributionUtilityService.distributionFee=-0.013251367801793017}

Discovering configurable properties

To fully take advantage of this scheme, it is necessary to somehow discover the full set of configurable properties in a set of modules. Otherwise, it remains necessary to somehow discover configurable properties by reading documentation. It is not enough to be able to save the combined configuration drawn from a set of configuration files, because this gives no guarantee of completeness or correctness.

Propagating model state from bootstrap to sim

Some models have significant state that affects their energy use behavior or storage capacity. Failure to propagate this state from a boot session to a sim session could result in significant discontinuities and make predictions unnecessarily difficult for brokers (see Issue #755). So it seems we also need a mechanism to treat bootstrap state as configuration.

Analysis of the situation (prior to a fix for #755):

  • The boot record is written in CompetitionSetupService.saveBootstrapData().
  • Currently, there are two clauses in the boot record. Tags are "config" and "bootstrap". Both are communicated to brokers. The config section includes a single clause, the Competition instance. The bootstrap section contains customer usage, market, and weather data.
  • In sim mode, data from the bootstrap record is read in two parts. The Competition instance is read in CompetitionSetupService.preGame(URL) and used to configure the current Competition instance and to initialize the clock. The bootstrap data is handled in processBootDataset(URL).
  • Module initialization happens before the boot record is processed, by the call to CSS.preGame() at the top of CSS.preGame(URL).

Alternative approaches:

Ideally modules would encapsulate the details of their state data, and would initialize themselves from the boot record during initialization. This would require that the module-initialization data be pulled from the boot record before initialization.

  • Initializable modules are already subclasses of InitializationService, which is currently an interface. If it were an abstract class, it could (potentially) provide access to the boot data. However, this would require that all existing modules be changed, and would break if any of them are already subclasses of some other class.
  • The initialize() method already includes a reference to the Competition instance. Perhaps access to bootstrap data could be passed in the Competition? However, that would clutter up a domain type that is shared with brokers.
  • A new BootstrapData service could be used, with the implementation in server-main. It would be handed at least the bootstrap-state clause before module initialization. Modules would then have to recover their bootstrap-state data from this service during module initialization.
  • Perhaps the existing annotation-based configuration scheme can be extended to support bootstrap state. It might be a simple as adding a configuration clause to the boot record. That clause would then be added to the system configuration data before initializing modules.

Implementing an annotation-based approach

This would be an extension of the existing annotation-based configuration scheme, in which bootstrap state is just another source of configuration input.

Saving bootstrap state:

  • State data must be recorded in the non-hierarchical format used by the configuration service, which is a "properties" format.
  • State data must be gathered at the end of a boot session in a manner similar to the existing ServerPropertiesService.publishConfiguration() method. The publishConfiguration() method is called by individual modules at the end of their respective initialize() methods. The cleanest way to gather state data would be to add an interface, perhaps BootstrapState, with a method getBootstrapState(). CSS.saveBootstrapData() would then call these methods within a "bootstrap-state" clause.
  • The ConfigurableValue annotation would need an additional feature called bootstrapState that, if true, would cause the value to be gathered up as part of the bootstrap state.
  • common.config.Configurator.gatherPublishedConfiguration() is used to gather config data for publication to brokers. It, or something very much like it, could presumably be used to gather state. However, it does not currently handle configured instances, which is how the cold-storage model is set up. It explicitly stops at the class name. We could use the ConfigurableInstance annotation, assuming it's inherited. That can be fixed by annotating the ConfigurableInstance annotation itself with the @Inherited meta-annotation.
  • Note that in the case of configured instances, it does not make sense for the instances themselves to implement the BootstrapState interface, because they are not Spring services and so Spring will not find them. Instead, whatever Spring entity creates the instances should also be responsible for implementing the interface and saving their state.

Restoring bootstrap state:

  • Data is in the bootstrap dataset in the form of an xml-serialized Properties instance.
  • Modules are initialized in CCS.runOnce(). The bootstrap state needs to be added to the config before this point, preferably in CSS since that's where it's gathered and that's where the configuration is built.
  • CCS.startSimSession() calls preGame(bootDataset) before calling CCS.runOnce().
  • Configured instances must not initialize state variables, because this will happen after the instance is configured. At best, a state variable can have a default value of null, and be initialized just in case the value is still null in the initialize() method.

Original (non-) design

As of version 0.1.0 (December 2011), the server is configured by reading several configuration files:

  1. The Spring config file powertac.xml, found in the classpath. In the development env, this is in server-main/src/main/resources. This file sets up Apache MQ, and injects configuration data into a number of specific class instances. For example, this is how the timeslotPhase value is set in the various service modules, thereby implementing the basic per-timeslot server activity cycle. This data clearly needs to be global to the server, rather than local to the individual services.
  2. A properties file server.properties, also on the classpath. This is a properties-format file that sets basic competition parameters such as timeslot length, bootstrap length, etc.
  3. A configuration file for household-customer.
  4. A configuration file for factored-customer.
  5. The bootstrap-data file.
  6. PluginConfig instances.

Clearly, neither the Spring context configuration nor simple Java properties files satisfies all our needs. Apache commons configuration comes closer, but by itself does not solve the duplication problem.