introduction.tex

\chapter{Introduction}
This thesis examines reinforcement learning algorithms in the domain of
electricity trade.  In this chapter the motivation for research into electric
power trade is explained, the problem under consideration is defined and the
principle research contributions are stated.

\section{Research Motivation}
% \section{Motivation for Electricity Market Research}
Quality of life for a person is directly proportional to his or her electricity
usage \cite{alam:qol}.  The world population is currently 6.7~billion and
forecast to exceed 9~billion by 2050 \cite{un:pop}. Electricity production
currently demands over one third of the annual primary energy extracted
\cite{iea:2010} and as more people endeavour to improve their quality of life,
finite fuel resources will become increasingly scarce. Market mechanisms, such
as auctions, where the final allocation is based upon the claimants' willingness
to pay for the goods, provide a device for efficient allocation of resources in
short supply.  In 1990 the UK became the first large industrialised country to
introduce competitive markets for electricity generation.

The inability to store electricity, once generated, in a commercially viable
quantity prevents it from being traded as a conventional commodity. Trading
mechanisms must allow shortfalls in electric energy to be purchased at short
notice from quickly dispatchable generators. Designed correctly, a competitive
electricity market can promote efficiency and drive down costs to the consumer,
while design errors can allow market power to be abused and market prices to
become elevated. It is necessary to research electricity market architectures to
ensure that their unique designs are fit for purpose.

The value of electricity to society makes it impractical to experiment with
radical changes to trading arrangements on real systems. The average total
demand for electricity in the United Kingdom (UK) is approximately 45GW and the
cost of buying 1MW for one hour is around \pounds40 \cite{decc:dukes09}. This
equates to yearly transaction values of \pounds16 billion.  The value of
electricity becomes particularly apparent when supply fails. The New York
black-out in August 2003 involved a loss of 61.8GW of power supply to
approximately 50 million consumers. The majority of supplies were restored
within two days, but the event is estimated to have cost more than \$6~billion
\cite{minkel:2008,icf:2003}.

An alternative approach is to study abstract mathematical models of markets with
sets of appropriate simplifying approximations and assumptions applied. Market
architecture characteristics and the consequences of proposed changes can be
established by simulating the models as digital computer programs. Competition
between participants is fundamental to all markets, but the strategies of humans
can be challenging to model mathematically. One available option is to use
reinforcement learning algorithms from the field of artificial intelligence.
These methods can be used to represent adaptive behaviour in competing players
and have been shown to be able to learn highly complex strategies
\cite{tesauro:gammon}.  This thesis makes advances in electricity market
participant modelling through the application of a genre of reinforcement
learning methods called policy gradient algorithms.

\section{Problem Statement}% and Hypothesis}%/Aims \& Objectives}
Individuals participating in an electricity market (be they representing
generating companies, load serving entities, firms of traders etc.)~must utilise
complex multi-dimensional data to their advantage. This data may be noisy,
sparse, corrupt, have a degree of uncertainty (e.g.~demand forecasts) or be
hidden from the participant (e.g.~competitor bids).  Reinforcement learning
algorithms must also be able to operate with data of this kind if they are to
successfully model participant strategies.

Traditional reinforcement learning methods, such as Q-learning, attempt to find
the \textit{value} of each available action in a given state.  When discrete
state and action spaces are defined, these methods become restricted by
Bellman's Curse of Dimensionality \cite{bellman:1961} and can not be readily
applied to complex problems.  Function approximation techniques, such as
artificial neural networks, can allow these methods to be applied to continuous
environment representations. However, value function approximation has been
shown to result in convergence issues, even in simple problems
\cite{tsitsiklis:94,peters:enac,gordon:95,baird:95}.

Policy gradient reinforcement learning methods do not attempt to approximate a
value function, but instead try to approximate a \textit{policy function} that,
given the current perceived state of the environment, returns an action
\cite{petersScholar}. They do not suffer from many of the problems that mar
value function based methods in high-dimensional problems.  They have strong
convergence properties, do not require that all states be continuously visited
and work with state and action spaces that are continuous, discrete or mixed
\cite{peters:enac}.  Policy performance may be degraded by uncertainty in state
data, but the learning methods do not need to be altered. They have been
successfully applied in many operational settings, including: robotic control
\cite{shaal:robots}, financial trading \cite{moody:direct} and network routing
\cite{peshkin:routing} applications.

It is proposed in this thesis that agents which learn using policy gradient
methods may outperform those using value function based methods in simulated
competitive electricity trade.  It is further proposed that policy gradient
methods may operate better under dynamic electric power system conditions,
achieving greater profit by exploiting constraints to their benefit. This thesis
will compare value function based and policy gradient learning methods in the
context of electricity trade to explore these proposals.

% Methods including learning classifier systems, genetic algorithms and
% reinforcement learning have been successfully used to research the
% charateristics of energy markets in the past\cite{weidlich:08}.  This an
% alternative to the traditional closed-form equilibrium approaches to game
% theory research in which behavior emerges from the interactions of many
% separable, self-serving agents. Typically in these studies, an agent is
% associated with a portfolio of generating units and/or dispatchable loads.
% Interaction with the environment involves submission of offers to sell or bids
% to buy\footnote{Beware that certain authors may use the term ``bid'' to refer
% to an offer to sell when discussing single-sided auctions.} a quantity of
% power at a specified price in a particular time period.  The learning
% algorithms typically use revenue or earnings as a reward signal and adjust the
% policy used to select offer/bid price and quantity values.

% Research into electricity markets using reinforcement learning typically
% involves discretization the action domain, often into incremental markups on
% marginal cost.  Also, either sensor domains are discretized or state
% information is disregarded altogether.  Despite these simplifications, authors
% have been drawn many practical conclusions from this approach
% \cite{weidlich:08}. If learning algorithms are to deliver on their potential
% for application in operational electricity market settings, modelling
% continuous domains will be necessary.

% Traditional reinforcement learning methods such as Sarsa and Q-learning (See
% Section \ref{sec:rl}, below) can be applied to systems with continuous domains
% by using connectionist systems for value function approximation
% \cite{barto:neuron}. However, feedback between policy updates and value
% function changes can result in oscillations or divergence in these algorithms
% even when applied to simple systems \cite{peters:enac}.  In response to this,
% policy-gradient methods, pioneered by Williams \cite{williams:reinforce},
% which search directly in the policy space have been developed and applied in
% many real-life settings
% \cite{barto:policy,shaal:robots,moody:direct,peshkin:routing}.  The proposal
% is to combine policy-gradient reinforcement learning methods with artificial
% neural networks for continuous policy function approximation and apply to
% simulations electricity trade.  The hypothesis being that superior use of
% sensor data results in improved performance over previously applied
% value-function methods.

\section{Research Contributions}
The research presented in this thesis pertains to the academic fields of
electrical power engineering, artificial intelligence and economics.  The
principle contributions made by this thesis are:
\begin{enumerate}
  \item The first application of policy gradient reinforcement learning
  methods in simulated electricity trade.  A class of unsupervised learning
  algorithms, designed for operation in multi-dimensional, continuous,
  uncertain and noisy environments, are applied in a series of dynamic
  techno-economic simulations.
  \item The first application of a non-linear AC optimal power flow formulation
  in agent based electricity market simulation.  Not applying the constraining
  assumptions of linearised DC models provides more accurate electric power
  systems models in which reactive power flows and voltage magnitude constraints
  are considered.
  \item A new Stateful Roth-Erev reinforcement learning method for application
  in complex environments with dynamic state.
  \item A comparison of policy gradient and value function based reinforcement
  learning methods in their convergence to states of Nash equilibrium. Results
  from published research for value function based methods are reproduced and
  extended to provide a foundation for the application of policy gradient
  methods in more complex electric power trade simulations.
  \item An examination of the exploitation of electric power system constraints
  by policy gradient reinforcement learning methods.  The
  superior multi-dimensional, continuous data handling abilities of policy
  gradient methods are tested by exploring their ability to observe voltage
  constraints and exploit them to achieve increased profits.
  \item An extensible open source multi-learning-agent-based
  power exchange auction market simulation framework for electric power trade
  research. Sharing software code can dramatically accelerate research of this
  kind and an extensive suite of the tools developed for this thesis has been
  released to the community.
  \item The concept of applying the Neuro-Fitted Q-Iteration and GQ$(\lambda)$
  methods in simulations of competitive energy trade. New unsupervised learning
  algorithms developed for operation in continuous environments could be
  utilised in electric power trade simulation and some of the most promising
  examples have been identified.
%   \item A formulation of optimal power flow as a reinforcement learning
%   problem.
%   \item Capital investment planning formulated as a reinforcement learning
%   problem.
%  \item A model of the UK transmission system derived from data in the
%  National Grid Seven Year Statement.
\end{enumerate}
The publications that have resulted from this thesis are:
%\citeA{lincoln:pyreto,lincoln:eem07,lincoln:upec}.

\vspace{1pc}
Lincoln, R., Galloway, S., Stephen, B., \& Burt, G. (Submitted for review).
Comparing Policy Gradient and Value Function Based Reinforcement
Learning Methods in Simulated Electrical Power Trade.
\ul{IEEE Transactions on Power Systems.}

\vspace{1pc}
Lincoln, R., Galloway, S., \& Burt, G. (2009, May 27-29). Open source,
agent-based energy market simulation with Python. In \ul{Proceedings of
the 6$^{\textrm{th}}$ International Conference on the European Energy Market,
2009. EEM 2009.} (p.~1-5).

\vspace{1pc}
Lincoln, R., Galloway, S., \& Burt, G. (2007, May 23-25). Unit
commitment and system stability under increased penetration of distributed
generation. In \ul{Proceedings of the 4$^{\textrm{th}}$ International Conference
on the European Energy Market, 2007. EEM 2007.} Krac\'ow, Poland.

\vspace{1pc}
Lincoln, R., Galloway, S., Burt, G., \& McDonald, J. (2006, 6-8).
Agent-based simulation of short-term energy markets for highly distributed
power systems. In \ul{Proceedings of the 41$^{\textrm{st}}$ International
Universities Power Engineering Conference, 2006. UPEC '06.} (Vol.~1,
p.~198-202).

\vspace{1pc}
This thesis also resulted in invitations to present at the tools sessions of the
Common Information Model (CIM) Users Group meetings in Genval, Belgium and
Charlotte, North Carolina, USA in 2009.

% This paper compares policy-gradient reinforcement learning algorithms
% REINFORCE and ENAC with value-function methods Roth-Erev, Q($\lambda$) and
% Sarsa in their relative ability to trade electricity competitively.  Power
% systems are modelled as balanced three-phase AC networks in the steady-state.
% Offers/bids for active and reactive power from agent participants are cleared
% using AC optimal power flow with an auction interface that returns single
% period revenue and earnings values. Through individual and multi-player
% experiments the methods are compared in their ability to learn quickly,
% compete in large systems and exploit characteristics of the power system. It
% is shown that, in electricity trade, policy-gradient methods: \begin{itemize}
% \item converge successfully on an optimal policy, \item are slower to converge
% than value-function algorithms, \item can learn more complicated
% characteristics of the power system than value-function algorithms, \item
% scale better when applied to larger systems and reactive power markets.
% \end{itemize}

% Section~\ref{sec:opf} of this paper presents the power system model, the
% optimal power flow formulation and the auction interface from MATPOWER.
% Reinforcement learning methods Sarsa, Q($\lambda$), Roth-Erev, REINFORCE and
% ENAC are defined in Section~\ref{sec:rl}. Section~\ref{ch:method} introduces
% the three experiments used to compare the aforementioned methods. Numerical
% results from these experiments are reported in Section \ref{ch:results} and an
% interpretation and critical analysis of them is given in
% Section~\ref{sec:discuss}.  Finally, a review of related research is presented
% in Section~\ref{sec:related} and Section~\ref{ch:conclusion} provides a
% conclusion.

% \section{Reader's Guide} In this thesis classic and modern reinforcement
% learning methods are applied in the domain of electric power trade.  The
% reader will require a certain degree of prior knowledge in these fields and
% may need to read Chapter \ref{ch:background} and much of the referenced
% material, to fully understand the methodology used.  This thesis is written
% for several kinds of readers.  A student who has taken an energy economics
% class or two may appreciate it as an introduction to electricity markets and
% their simulation.  Research students embarking upon postgraduate study of
% electricity markets may find the ideas for further work in Chapter
% \ref{sec:furtherwork} of particular interest. Researchers experienced in
% adaptive control and machine learning, looking for new application domains for
% their methods, may find the electricity market model definition in Chapter
% \ref{ch:method} to be of value.

\section{Thesis Outline}
This thesis is organised into nine chapters.  Chapter \ref{ch:background}
provides background information on electricity supply, wholesale electricity
markets and reinforcement learning.  It describes how optimal power flow
formulations can be used to model electricity markets and defines the
reinforcement learning algorithms that are later compared. The chapter is
intended to enable readers unfamiliar with this field of research to understand
the techniques used in the subsequent chapters.

In Chapter \ref{ch:related_work} the research in this thesis is described in the
context of previous work related in terms of application field and methodology.
Publications on agent based electricity market simulation are reviewed with
emphasis on the participant behavioural models used. Previous applications of
policy gradient learning methods in other types of market setting are also
covered.  The chapter illustrates the trend in this field towards more complex
participant behavioural models and highlights some of the gaps in the existing
research that this thesis aims to fill.
% Finally, the contribution made by releasing the source code developed for this
% thesis as open source is shown in a reiview of the main open source electric
% power Engineering software tools.

Chapter \ref{ch:method} describes the power exchange auction market model and
the multi-agent system used to simulate electricity trade.  It defines the
association of learning agents with portfolios of generators, the process of
offer submission and the reward process. The chapter describes the components
that are common to the specific simulations that are then described.
% Finally, it explains how look-up tables, used with value function based
% methods, and artificial neural networks, used for policy function
% approximation, are are structured.

Simulations that examine the convergence to a Nash equilibrium of systems of
multiple electric power trading agents are reported in Chapter
\ref{ch:nashanalysis}. A six bus test case is used and results for four learning
algorithms under two cost configurations are presented and discussed. The
chapter confirms that policy gradient methods can be used in electric power
trade simulations, in the same way as value function based methods and provides
a foundation for their application in more complex experiments.

Chapter \ref{ch:exploitation} examines the ability of agents to learn policies
for exploiting constraints in simulated power systems.  The 24 bus model from
the IEEE Reliability Test System provides a complex environment with dynamic
loading conditions. The chapter is used to determine if the multi-dimensional
continuous data handling abilities of policy gradient methods can be exploited
by agents to learn more complex electricity trading policies than those
operating in discrete trading environment representations.

The primary conclusions drawn from the results in this thesis are summarised in
Chapter \ref{ch:conclusion}.  Shortcomings of the approach are noted and the
broader implications are addressed.  Some ideas for further work are also
outlined, including alternative reinforcement learning methods and potential
applications for a UK transmission system model developed for this thesis.

% Free market democracy has underpinned the transformation of large western
% economies in the post war era and continues to be relied upon.  Towards the
% end of the nineteenth century the principals of competitive trade were
% successfully applied to electric power industries, beginning in the UK in
% March 1990 with the creation of The Electricity Pool.

% Two trends characterise modern power systems Engineering: Increased
% liberalisation of the industry through competitive energy trade and increased
% presence of renewable energy generation on the network.  As the number and
% variety of electricity sources becomes greater, the necessity for automated
% trade of their power increases.  Control algorithms may draw sensor input from
% data networks and other sources and use some relevant measure of performance,
% such as profitability, to learn from trading decisions.

% \IEEEPARstart{E}{nergy} consumers will often not switch suppliers despite
% apparent benefits of doing so, or mistakenly switch to more expensive
% contracts[].  Energy companies are thus unlikely to reduce prices as this can
% not be expected to result in new contracts.  One possible solution to this
% problem is to give the responsibility for buying energy to autonomous,
% learning control algorithms.  Drawing information from the internet and using
% standardised messaging [CIM] to form contracts, energy bots have the potential
% to introduce greater liquidity to energy markets and promote reduction of
% costs to the consumer.

% \IEEEPARstart{D}{ynamic} competitive markets will be key in the transition to
% a low carbon economy\cite{decc:transition}.  Fluctuations in global oil prices
% have recently raised questions over the functioning of the energy market.
% Energy industries in the UK contribute 4.8\% of 2.13 trillion dollar gross
% domestic product.  The UK electricity industry supplies \~26 million customers
% with \~350TWh of energy each year at a cost of 3.5--9.0 p/kWh. The New York
% blackout in August, 2003 resulted in the loss of 61.8 GW of electric load
% associated with 50 million consumers.  Supply by was largely restored within
% two days and the event is estimated to have cost \$7--\$10 billion.  Small
% improvements in the design of markets associated with this industry can have
% an impact greatly upon the welfare of society and conversely also. To
% understand the complex dynamics of these systems it is possible to simulate
% them computationally.  One alternative to the game theoretic models typically
% employed in computational economics is the study of emergent behaviour in
% collections of individual autonomous actors.  Representing competitive
% behaviour is key to the simulation of markets in this way. Simple heurisitc
% approaches have been taken in the study of \ldots[ConzelmannEMCAS].  While
% more complex machine learning techniques such as state vector machines have
% been employed in the study of market efficiency and market power[Bunn,
% Bagnall].  Reinforcement Learning (RL) is an unsupervised machine learning
% technique, most commonly applied to control problems, and as such is well
% suited to this application [Acrobot]\cite{suttonbarto:reinforcement}. [TD()].
% In this paper RL is used to control the output and price of generating units
% and despatchable loads connected in a balanced three-phase high-voltage power
% system. RL has been applied before in a very similar manner to the energy
% trading problem using the Roth-Erev algorithm[TesfatsoRE].  The software
% implementation for this algorithm used in the simulations for this paper was
% translated from[TesfatsoAMES].

% Engineers must strive for complexity in their work.  Rarely will a simple
% solution will perform a function to a higher degree than a more complex one.
% Certainly, where a function is either performed or not performed, prefer the
% simpler one, but most often problems can be solved to varying degrees.

% The broad aim of the research presented in this thesis is to prove that the
% above conjecture applies to reinforcement learning algorithms for power trade.
% Previous research in this field (See Chapter~\ref{ch:related_work} below) has
% used very simple algorithms in relation to those from the latest advances in
% artificial intelligence (See Sections~\ref{sec:enac}~and~\ref{sec:reinforce}
% below).  The goal is to prove that policy gradient methods, using artificial
% neural networks for policy function approximation, are better suited to
% learning the complex dynamics of a power system.

% Industrialised societies have become increasingly reliant on the supply of
% electric energy since the connection of large power stations began in 1938.
% The extent to which this is true can be seen in the financial impact that loss
% of supply has on society.
%
%
% In June 2004 the United Kingdom (UK) became a net importer of natural gas for
% the first time in 8 years[EIA DOE].  Since energy industry privatisation by
% the Conservatives in the 1980s, use of domestic gas reserves for electricity
% generation has been encouraged and exploited.
%
% % Insert dash for gas pie-chart here.
%
% As UK natural gas production has now peaked and consumption continues to grow,
% concern over reliance on imports from less stable regions has increased.
%
% The UK is hugely reliant on fossil fuels.  More than three quarters of UK
% electricity is generated from a relatively small number of large coal and gas
% fired power stations[Energy Digest].  Of the 298 stations with capacity over
% 1MW in the UK, 63 gas fuelled (including CCGT) and 13 coal fired stations
% supply over 290 TWh of the UK's 393TWh annual electric energy
% production[DUKES].  Much of the remainder is generated through nuclear
% fission.
%
%
% Concerns over climate change and security of supply have caused the UK
% government to pursue self-sustainable sources of electric energy.  This is
% illustrated by the government?s recent decision to make a legally binding
% commitment to an 80\% reduction in carbon dioxide emissions by 2050, relative
% to 1990 levels[].
%
%
% Relative to most other commodities, trade of electric energy is still in its
% infancy.  Liberalisation and unbundling of electricity supply industries costs
% many millions of pounds to implement[].  Countries, having made this
% investment, continue to restructure and adjust their energy markets in the
% hope of further reducing costs to the consumer and promoting innovation and
% efficiency through competition.
%
%
% \section{Conventional national power systems}
% Economies of scale prompted construction of the first large-scale power
% stations in the UK at the beginning of the twentieth century.  Following the
% introduction of the Electricity (Supply) Act 1926 the largest and most
% efficient of these were connected by a series of regional high-voltage
% three-phase AC grids synchronised at 50 Hertz.  Integration was completed and
% a national transmission system made operational in the UK for the first time
% in 1938.  This approach to electricity supply is largely the same as that
% still employed throughout the UK to the present day.  Alternating current,
% mainly from rotating synchronous machines, is transformed to high voltages for
% bulk transmission over long distances with high efficiency.  Power from the
% transmission system is fed through distribution networks in a uni-directional
% fashion at lower voltages before final usage.
%
% While maintenance and extension of the transmission system and of distribution
% networks is an everyday activity for energy utilities, many power system
% components have extremely long operational lifetimes[].  This and the
% magnitude of the capital investment made post-war in construction of the
% electricity networks suggests that there is likely to be little change in the
% topology of the system of wires in the foreseeable future.  This is further
% compounded by the fact that distribution networks are often radial in their
% structure.  Reliability and protection are major issues in the operation of
% power systems and the task of detecting and isolating a fault is often more
% difficult in systems that are meshed.
%
% Large-scale thermal power stations operate steam and gas turbines around 13
% metres in length, approximately 400 tons in weight and rotate at up to 3600
% revolutions per minute[SIEMENS].  The kinetic energy stored in these turbines
% and the connected synchronous machines is vast and the associated flywheel
% effect plays an important role in smoothing short-term imbalances in supply
% and demand[].
%
% Synchronous machines used in large thermal power stations invariably use a
% rotor winding that is excited and a magnetic flux created by a DC current.
% Controlling the magnitude of the rotor field current allows the reactive power
% provided or absorbed by the machine to be adjusted.  Reactive power is
% essential in maintaining system voltage within permissable limits[].
%
% \section{Highly distributed national power systems}
% The highly distributed power system is a conceptual future for the UK's
% national energy network.  It is capable of meeting current targets for reduced
% greenhouse gas emissions[] and reduces reliance on foreign fuel imports.  It
% is a system in which all but the least polluting fossil fuelled power stations
% have been decommissioned.  Their output supplanted by distributed generation,
% supported with demand-side management measures and advances in energy storage
% technology.
%
% Distributed (or embedded) generation is most easily defined as electricity
% production plant connected to the national electricity network at the
% distribution level[DG Book].  The transmission system being defined as that
% which operates at of above 275kV in England and Wales or at or above 132kV in
% Scotland.  This encompasses most small-scale plant, but there exist exceptions
% such as large-scale wind farms.
%
% Hydro-electric dams, biomass fuelled power stations and wind farms are the
% three sources of renewable energy currently available in the UK with
% capacities equivalent to that of fossil fuelled plant.  Hydro-electric power
% stations are often large-scale and transmission connected for bulk transport
% of power to load centres.  The low energy density of biomass fuels, relative
% to that of fossil fuels, often dictates that related generating plant is
% located close the fuel source origin and connected to a distribution network.
% The time varying nature of wind energy necessitates the use of induction
% machines for generation.  These are typically sinks of reactive power and
% require support for operation and maintenance of voltage.
%
% \subsection{Increased source granularity}
% In terms of demand the UK power system is already highly distributed.  The
% number of generators, controllable loads and storage systems is expected to be
% much greater in a HDPS.  However, there are at present already around 260
% generators supplying energy to consumers and the mechanisms currently in place
% to facilitate the trade of their power may be adaptable.
%
% Consumers and owners of small-scale generation will continue to desire
% protection from the risks associated with the wholesale marketplace.  Today?s
% typical domestic and commercial supply contracts between consumers and energy
% retailers offer such protection.  As the penetration of DER grows the role of
% energy retailers offering aggregation services will likely grow as more plant
% owners come to desire representation in the marketplace.  While there are a
% great many ways in which this aggregation may be arranged it remains
% essentially a contractual arrangement. A much larger challenge lies in the
% operation and control of a system in which the state of plant is principally
% determined by a decentralised free market.  The network may be divided
% into ?cells? for the provision of a single point of control, but each cell may
% contain individual items of plant being aggregated by different companies.
% Controlling the DER, so as to adhere to system constraints, must be done in
% the most economically efficient, in the least environmentally damaging manner
% and according to contracted access arrangements.  Also, the details of any
% control measures undertaken must be fed back to the marketplace such that they
% may be taken into consideration during the settlement process.  This interface
% between the technical domain and the commercial mechanisms remains an open
% research topic.
%
% \subsection{Inversion of control}
% Consumers have grown accustomed to using power from the grid at will.  System
% loads are by and large passive and, with the exception of dual rate white
% metered loads, only the largest make adjustments to their consumption
% according to price signals or system conditions.  It is likely that the
% fluctuating nature of the power output from generators exploiting renewable
% energy sources will necessitate an increase in the adoption of active loads.
% As control shifts from being almost purely supply-side, an opportunity opens
% for the energy marketplace to offer appropriate consumer choice, not only in
% terms of cost, but Quality of Service (QoS) also. Advances in smart metering
% promise to support such a migration.
%
% \subsection{Reduced network reticulation}
% The connection of controllable energy resources to lower voltage and less
% reticulated areas of the network may also offer new possibilities for
% structuring the relationship between the market and the system for management
% of network constraints.  There are generally three options for power system
% constraint management.
%
% \begin{itemize}
%   \item Only permit the formation of energy trades if delivery is physically
%   feasible.
%   \item Impose delivery charges which increase as network constraints are
%    approached.
%   \item Request extended bids and offers which include costs associated with
%   the adjustment of participant?s desired position.
% \end{itemize}
%
% The third option most closely describes the method currently used in the
% United Kingdom.  In a highly reticulated network it is difficult to determine
% the direction of each participant?s energy flows.  In turn, this poses
% difficulties in determining if a particular delivery is feasible and which
% participants are responsible for congesting the network.  Distribution
% networks are typically less reticulated than the transmission network to which
% the majority of generation is connected at present.  Therefore, there may be
% opportunities to utilise options one or two in an energy marketplace for an
% HDPS.
%
% Furthermore, the lower voltage of distribution networks may open up
% opportunities for more widespread use of power electronics technologies such
% as FACTS and phase shifting devices.  These would provide a limited ability to
% direct the flow of electrical energy.  How this might be managed on a wide
% scale and how efficient interaction with the marketplace might be achieved is
% open to investigation
%
% \subsection{Dual objective optimisation}
% Along with concern over the UK's dependence on natural gas imports, concern
% over the environmental impact of electricity generation is a primary motivator
% for a move to HDPS.  The energy market is expected to simultaneously minimise
% costs to the consumer and encourage reduced emission of greenhouse gasses.
%
% The European Union Greenhouse Gas Emission Trading Scheme (EU ETS) is an
% example of how a second marketplace running parallel to electricity trade, in
% this case trading allowances, may give weight to this new objective.  There
% are other ways in which the greenhouse gas output of certain technologies may
% be taken into consideration and this remains an open and important research
% topic.