2019-05-23-meta-modelling-and-deploying-ml-software.html

---
title: "Meta-Modelling and Deploying ML Software"
venue: "The Mathematics of Deep Learning and Data Science"
abstract: "Data is not so much the new oil, it is the new software. Data driven algorithms are increasingly present in continuously deployed production software. What challenges does this present and how can the mathematical sciences help?"
author:
- given: Neil D.
  family: Lawrence
  url: http://inverseprobability.com
  institute: Amazon Cambridge and University of Sheffield
  twitter: lawrennd
  gscholar: r3SJcvoAAAAJ
  orcid: 
edit_url: https://github.com/lawrennd/talks/edit/gh-pages/_ml/meta-modelling-and-deploying-ml-software.md
date: 2019-05-23
published: 2019-05-23
week: 0
reveal: 2019-05-23-meta-modelling-and-deploying-ml-software.slides.html
edit_url: https://github.com/lawrennd/talks/edit/gh-pages/_ml/meta-modelling-and-deploying-ml-software.md
layout: talk
categories:
- notes
---



<!-- Do not edit this file locally. -->
<!---->
<!-- Do not edit this file locally. -->
<!-- Do not edit this file locally. -->
<!-- The last names to be defined. Should be defined entirely in terms of macros from above-->
<!--

-->
<h1 id="introduction">Introduction</h1>
<h2 id="peppercorns">Peppercorns</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_ai/includes/peppercorn.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_ai/includes/peppercorn.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<div class="figure">
<div id="peppercorn-siri-figure" class="figure-frame">
<iframe width="600" height="450" src="https://www.youtube.com/embed/1y2UKz47gew?start=" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen>
</iframe>
</div>
<div id="peppercorn-siri-magnify" class="magnify" onclick="magnifyFigure(&#39;peppercorn-siri&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="peppercorn-siri-caption" class="caption-frame">
<p>Figure: A peppercorn is a system design failure which is not a bug, but a conformance to design specification that causes problems when the system is deployed in the real world with mischevious and adversarial actors.</p>
</div>
</div>
<p>Asking Siri “What is a trillion to the power of a thousand minus one?” leads to a 30 minute response<a href="#fn1" class="footnote-ref" id="fnref1" role="doc-noteref"><sup>1</sup></a> consisting of only 9s. I found this out because my nine year old grabbed my phone and did it. The only way to stop Siri was to force closure. This is an interesting example of a system feature that’s <em>not</em> a bug, in fact it requires clever processing from Wolfram Alpha. But it’s an unexpected result from the system performing correctly.</p>
<p>This challenge of facing a circumstance that was unenvisaged in design but has consequences in deployment becomes far larger when the environment is uncontrolled. Or in the extreme case, where actions of the intelligent system effect the wider environment and change it.</p>
<p>These unforseen circumstances are likely to lead to need for much more efficient turn-around and update for our intelligent systems. Whether we are correcting for security flaws (which <em>are</em> bugs) or unenvisaged circumstantial challenges: an issue I’m referring to as <em>peppercorns</em>. Rapid deployment of system updates is required. For example, Apple have “fixed” the problem of Siri returning long numbers.</p>
<p>Here’s another one from Reddit, of a Tesla Model 3 system hallucinating traffic lights.</p>
<iframe id="reddit-embed" width="600" height="450" src="https://www.redditmedia.com/r/teslamotors/comments/nrs8kf/you_think_ice_cream_truck_stop_signs_are_a_problem/?ref_source=embed&amp;ref=share&amp;embed=true" sandbox="allow-scripts allow-same-origin allow-popups" frameborder="0" scrolling="no">
</iframe>
<p>The challenge is particularly acute because of the <em>scale</em> at which we can deploy AI solutions. This means when something does go wrong, it may be going wrong in billions of households simultaneously.</p>
<p>You can also check this blog post on <a href="http://inverseprobability.com/2017/11/15/decision-making">Decision Making and Diversity</a>. and this blog post on <a href="http://inverseprobability.com/2018/02/06/natural-and-artificial-intelligence">Natural vs Artifical Intelligence</a>..</p>
<h1 id="deep-learning">Deep Learning</h1>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_ml/includes/deep-learning-overview.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_ml/includes/deep-learning-overview.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<!-- No slide titles in this context -->
<h2 id="deepface">DeepFace</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_ml/includes/deep-face.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_ml/includes/deep-face.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<div class="figure">
<div id="deep-face-figure" class="figure-frame">
<div class="centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//deepface_neg.png" width="100%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="deep-face-magnify" class="magnify" onclick="magnifyFigure(&#39;deep-face&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="deep-face-caption" class="caption-frame">
<p>Figure: The DeepFace architecture <span class="citation" data-cites="Taigman:deepface14">(Taigman et al., 2014)</span>, visualized through colors to represent the functional mappings at each layer. There are 120 million parameters in the model.</p>
</div>
</div>
<p>The DeepFace architecture <span class="citation" data-cites="Taigman:deepface14">(Taigman et al., 2014)</span> consists of layers that deal with <em>translation</em> and <em>rotational</em> invariances. These layers are followed by three locally-connected layers and two fully-connected layers. Color illustrates feature maps produced at each layer. The neural network includes more than 120 million parameters, where more than 95% come from the local and fully connected layers.</p>
<h3 id="deep-learning-as-pinball">Deep Learning as Pinball</h3>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_ml/includes/deep-learning-as-pinball.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_ml/includes/deep-learning-as-pinball.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<div class="figure">
<div id="early-pinball-figure" class="figure-frame">
<div class="centered centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//576px-Early_Pinball.jpg" width="50%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="early-pinball-magnify" class="magnify" onclick="magnifyFigure(&#39;early-pinball&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="early-pinball-caption" class="caption-frame">
<p>Figure: Deep learning models are composition of simple functions. We can think of a pinball machine as an analogy. Each layer of pins corresponds to one of the layers of functions in the model. Input data is represented by the location of the ball from left to right when it is dropped in from the top. Output class comes from the position of the ball as it leaves the pins at the bottom.</p>
</div>
</div>
<p>Sometimes deep learning models are described as being like the brain, or too complex to understand, but one analogy I find useful to help the gist of these models is to think of them as being similar to early pin ball machines.</p>
<p>In a deep neural network, we input a number (or numbers), whereas in pinball, we input a ball.</p>
<p>Think of the location of the ball on the left-right axis as a single number. Our simple pinball machine can only take one number at a time. As the ball falls through the machine, each layer of pins can be thought of as a different layer of ‘neurons.’ Each layer acts to move the ball from left to right.</p>
<p>In a pinball machine, when the ball gets to the bottom it might fall into a hole defining a score, in a neural network, that is equivalent to the decision: a classification of the input object.</p>
<p>An image has more than one number associated with it, so it is like playing pinball in a <em>hyper-space</em>.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="op">%</span>pip install pods</span></code></pre></div>
<div class="figure">
<div id="pinball-initialization-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//pinball001.svg" width="80%" style=" ">
</object>
</div>
<div id="pinball-initialization-magnify" class="magnify" onclick="magnifyFigure(&#39;pinball-initialization&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="pinball-initialization-caption" class="caption-frame">
<p>Figure: At initialization, the pins, which represent the parameters of the function, aren’t in the right place to bring the balls to the correct decisions.</p>
</div>
</div>
<div class="figure">
<div id="pinball-trained-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//pinball002.svg" width="80%" style=" ">
</object>
</div>
<div id="pinball-trained-magnify" class="magnify" onclick="magnifyFigure(&#39;pinball-trained&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="pinball-trained-caption" class="caption-frame">
<p>Figure: After learning the pins are now in the right place to bring the balls to the correct decisions.</p>
</div>
</div>
<p>Learning involves moving all the pins to be in the correct position, so that the ball ends up in the right place when it’s fallen through the machine. But moving all these pins in hyperspace can be difficult.</p>
<p>In a hyper-space you have to put a lot of data through the machine for to explore the positions of all the pins. Even when you feed many millions of data points through the machine, there are likely to be regions in the hyper-space where no ball has passed. When future test data passes through the machine in a new route unusual things can happen.</p>
<p><em>Adversarial examples</em> exploit this high dimensional space. If you have access to the pinball machine, you can use gradient methods to find a position for the ball in the hyper space where the image looks like one thing, but will be classified as another.</p>
<p>Probabilistic methods explore more of the space by considering a range of possible paths for the ball through the machine. This helps to make them more data efficient and gives some robustness to adversarial examples.</p>
<h2 id="containerization">Containerization</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_supply-chain/includes/containerisation.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_supply-chain/includes/containerisation.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<div class="figure">
<div id="container-2539942_1920-figure" class="figure-frame">
<div class="centered centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//supply-chain/container-2539942_1920.jpg" width="80%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="container-2539942_1920-magnify" class="magnify" onclick="magnifyFigure(&#39;container-2539942_1920&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="container-2539942_1920-caption" class="caption-frame">
<p>Figure: The container is one of the major drivers of globalization, and arguably the largest agent of social change in the last 100 years. It reduces the cost of transportation, significantly changing the appropriate topology of distribution networks. The container makes it possible to ship goods halfway around the world for cheaper than it costs to process those goods, leading to an extended distribution topology.</p>
</div>
</div>
<p>Containerization has had a dramatic effect on global economics, placing many people in the developing world at the end of the supply chain.</p>
<div class="figure">
<div id="wild-alaskan-cod-figure" class="figure-frame">
<table>
<tr>
<td width="45%">
<div class="centered centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//supply-chain/wild-alaskan-cod.jpg" width="90%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</td>
<td width="45%">
<div class="centered centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//supply-chain/wild-alaskan-cod-made-in-china.jpg" width="90%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</td>
</tr>
</table>
</div>
<div id="wild-alaskan-cod-magnify" class="magnify" onclick="magnifyFigure(&#39;wild-alaskan-cod&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="wild-alaskan-cod-caption" class="caption-frame">
<p>Figure: Wild Alaskan Cod, being solid in the Pacific Northwest, that is a product of China. It is cheaper to ship the deep frozen fish thousands of kilometers for processing than to process locally.</p>
</div>
</div>
<p>For example, you can buy Wild Alaskan Cod fished from Alaska, processed in China, sold in North America. This is driven by the low cost of transport for frozen cod vs the higher relative cost of cod processing in the US versus China. Similarly, <a href="https://www.telegraph.co.uk/news/uknews/1534286/12000-mile-trip-to-have-seafood-shelled.html" target="_blank">Scottish prawns are also processed in China for sale in the UK.</a></p>
<div class="figure">
<div id="environmental-impact-of-food-by-life-cycle-figure" class="figure-frame">
<div class="centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//supply-chain/environmental-impact-of-food-by-life-cycle.png" width="70%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="environmental-impact-of-food-by-life-cycle-magnify" class="magnify" onclick="magnifyFigure(&#39;environmental-impact-of-food-by-life-cycle&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="environmental-impact-of-food-by-life-cycle-caption" class="caption-frame">
<p>Figure: The transport cost of most foods is a very small portion of the total cost. The exception is if foods are air freighted. Source: <a href="https://ourworldindata.org/food-choice-vs-eating-local" class="uri">https://ourworldindata.org/food-choice-vs-eating-local</a> by Hannah Ritche CC-BY</p>
</div>
</div>
<p>This effect on cost of transport vs cost of processing is the main driver of the topology of the modern supply chain and the associated effect of globalization. If transport is much cheaper than processing, then processing will tend to agglomerate in places where processing costs can be minimized.</p>
<p>Large scale global economic change has principally been driven by changes in the technology that drives supply chain.</p>
<p>So many examples in terms of the need for intelligent decision making are based around the challenge of moving goods/energy/compute/water/medicines/drivers/people from where it is to where it needs to be. In other words matching supply with demand. That led me to a motto I developed while working in Amazon’s supply chain.</p>
<blockquote>
<p>Solve Supply Chain, then solve everything else.</p>
</blockquote>
<h2 id="statistical-emulation">Statistical Emulation</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/emulation.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/emulation.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<div class="figure">
<div id="met-office-unified-model-figure" class="figure-frame">
<div class="centered" style="">
<img class="negate" src="https://inverseprobability.com/slides/diagrams//simulation/unified_model_systems_13022018_1920.png" width="60%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="met-office-unified-model-magnify" class="magnify" onclick="magnifyFigure(&#39;met-office-unified-model&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="met-office-unified-model-caption" class="caption-frame">
<p>Figure: The UK Met office runs a shared code base for its simulations of climate and the weather. This plot shows the different spatial and temporal scales used.</p>
</div>
</div>
<p>In many real-world systems, decisions are made through simulating the environment. Simulations may operate at different granularities. For example, simulations are used in weather forecasts and climate forecasts. Interestingly, the UK Met office uses the same code for both, it has a <a href="https://www.metoffice.gov.uk/research/approach/modelling-systems/unified-model/index">“Unified Model” approach</a>, but they operate climate simulations at greater spatial and temporal resolutions.</p>
<div class="figure">
<div id="statistical-emulation-1-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//uq/statistical-emulation000.svg" width="80%" style=" ">
</object>
</div>
<div id="statistical-emulation-1-magnify" class="magnify" onclick="magnifyFigure(&#39;statistical-emulation-1&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="statistical-emulation-1-caption" class="caption-frame">
<p>Figure: Real world systems consist of simulators that capture our domain knowledge about how our systems operate. Different simulators run at different speeds and granularities.</p>
</div>
</div>
<div class="figure">
<div id="statistical-emulation-2-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//uq/statistical-emulation001.svg" width="80%" style=" ">
</object>
</div>
<div id="statistical-emulation-2-magnify" class="magnify" onclick="magnifyFigure(&#39;statistical-emulation-2&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="statistical-emulation-2-caption" class="caption-frame">
<p>Figure: A statistical emulator is a system that reconstructs the simulation with a statistical model.</p>
</div>
</div>
<p>A statistical emulator is a data-driven model that learns about the underlying simulation. Importantly, learns with uncertainty, so it ‘knows what it doesn’t know.’ In practice, we can call the emulator in place of the simulator. If the emulator ‘doesn’t know,’ it can call the simulator for the answer.</p>
<div class="figure">
<div id="statistical-emulation-5-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//uq/statistical-emulation004.svg" width="80%" style=" ">
</object>
</div>
<div id="statistical-emulation-5-magnify" class="magnify" onclick="magnifyFigure(&#39;statistical-emulation-5&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="statistical-emulation-5-caption" class="caption-frame">
<p>Figure: A statistical emulator is a system that reconstructs the simulation with a statistical model. As well as reconstructing the simulation, a statistical emulator can be used to correlate with the real world.</p>
</div>
</div>
<p>As well as reconstructing an individual simulator, the emulator can calibrate the simulation to the real world, by monitoring differences between the simulator and real data. This allows the emulator to characterize where the simulation can be relied on, i.e., we can validate the simulator.</p>
<p>Similarly, the emulator can adjudicate between simulations. This is known as <em>multi-fidelity emulation</em>. The emulator characterizes which emulations perform well where.</p>
<p>If all this modelling is done with judicious handling of the uncertainty, the <em>computational doubt</em>, then the emulator can assist in desciding what experiment should be run next to aid a decision: should we run a simulator, in which case which one, or should we attempt to acquire data from a real-world intervention.</p>
<h2 id="uncertainty-quantification">Uncertainty Quantification</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/uq-intro.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/uq-intro.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<blockquote>
<p>Uncertainty quantification (UQ) is the science of quantitative characterization and reduction of uncertainties in both computational and real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known.</p>
</blockquote>
<p>We will to illustrate different concepts of <a href="https://en.wikipedia.org/wiki/Uncertainty_quantification">Uncertainty Quantification</a> (UQ) and the role that Gaussian processes play in this field. Based on a simple simulator of a car moving between a valley and a mountain, we are going to illustrate the following concepts:</p>
<ul>
<li><p><strong>Systems emulation</strong>. Many real world decisions are based on simulations that can be computationally very demanding. We will show how simulators can be replaced by <em>emulators</em>: Gaussian process models fitted on a few simulations that can be used to replace the <em>simulator</em>. Emulators are cheap to compute, fast to run, and always provide ways to quantify the uncertainty of how precise they are compared the original simulator.</p></li>
<li><p><strong>Emulators in optimization problems</strong>. We will show how emulators can be used to optimize black-box functions that are expensive to evaluate. This field is also called Bayesian Optimization and has gained an increasing relevance in machine learning as emulators can be used to optimize computer simulations (and machine learning algorithms) quite efficiently.</p></li>
<li><p><strong>Multi-fidelity emulation methods</strong>. In many scenarios we have simulators of different quality about the same measure of interest. In these cases the goal is to merge all sources of information under the same model so the final emulator is cheaper and more accurate than an emulator fitted only using data from the most accurate and expensive simulator.</p></li>
</ul>
<h2 id="mountain-car-simulator">Mountain Car Simulator</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-simulation.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-simulation.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>To illustrate the above mentioned concepts we use the <a href="https://github.com/openai/gym/wiki/MountainCarContinuous-v0">mountain car simulator</a>. This simulator is widely used in machine learning to test reinforcement learning algorithms. The goal is to define a control policy on a car whose objective is to climb a mountain. Graphically, the problem looks as follows:</p>
<div class="figure">
<div id="mountain-car-figure" class="figure-frame">
<div class="centered" style="">
<img class="negate" src="https://inverseprobability.com/slides/diagrams//uq/mountaincar.png" width="60%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="mountain-car-magnify" class="magnify" onclick="magnifyFigure(&#39;mountain-car&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="mountain-car-caption" class="caption-frame">
<p>Figure: The mountain car simulation from the Open AI gym.</p>
</div>
</div>
<p>The goal is to define a sequence of actions (push the car right or left with certain intensity) to make the car reach the flag after a number <span class="math inline">\(T\)</span> of time steps.</p>
<p>At each time step <span class="math inline">\(t\)</span>, the car is characterized by a vector <span class="math inline">\(\mathbf{ x}_{t} = (p_t,v_t)\)</span> of states which are respectively the the position and velocity of the car at time <span class="math inline">\(t\)</span>. For a sequence of states (an episode), the dynamics of the car is given by</p>
<p><span class="math display">\[
\mathbf{ x}_{t+1} = f(\mathbf{ x}_{t},\textbf{u}_{t})
\]</span></p>
<p>where <span class="math inline">\(\textbf{u}_{t}\)</span> is the value of an action force, which in this example corresponds to push car to the left (negative value) or to the right (positive value). The actions across a full episode are represented in a policy <span class="math inline">\(\textbf{u}_{t} = \pi(\mathbf{ x}_{t},\theta)\)</span> that acts according to the current state of the car and some parameters <span class="math inline">\(\theta\)</span>. In the following examples we will assume that the policy is linear which allows us to write <span class="math inline">\(\pi(\mathbf{ x}_{t},\theta)\)</span> as</p>
<h2 id="mountain-car-set-up">Mountain Car Set Up</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-setup.py" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-setup.py', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>To run the mountain car example we need to install a python file that we’ll download.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> urllib.request</span></code></pre></div>
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a>urllib.request.urlretrieve(<span class="st">&#39;https://raw.githubusercontent.com/lawrennd/talks/gh-pages/mountain_car.py&#39;</span>,<span class="st">&#39;mountain_car.py&#39;</span>)</span></code></pre></div>
<p>And to render the environment, the <code>pyglet</code> library.</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="op">%</span>pip install pyglet</span></code></pre></div>
<p><span class="math display">\[
\pi(\mathbf{ x},\theta)= \theta_0 + \theta_p p + \theta_vv.
\]</span> For <span class="math inline">\(t=1,\dots,T\)</span> now given some initial state <span class="math inline">\(\mathbf{ x}_{0}\)</span> and some some values of each <span class="math inline">\(\textbf{u}_{t}\)</span>, we can <strong>simulate</strong> the full dynamics of the car for a full episode using <a href="https://gym.openai.com/envs/">Gym</a>. The values of <span class="math inline">\(\textbf{u}_{t}\)</span> are fully determined by the parameters of the linear controller.</p>
<p>After each episode of length <span class="math inline">\(T\)</span> is complete, a reward function <span class="math inline">\(R_{T}(\theta)\)</span> is computed. In the mountain car example, the reward is computed as 100 for reaching the target of the hill on the right hand side, minus the squared sum of actions (a real negative to push to the left and a real positive to push to the right) from start to goal. Note that our reward depends on <span class="math inline">\(\theta\)</span> as we make it dependent on the parameters of the linear controller.</p>
<h2 id="emulate-the-mountain-car">Emulate the Mountain Car</h2>
<div class="sourceCode" id="cb5"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="op">%</span>pip install gym</span></code></pre></div>
<div class="sourceCode" id="cb6"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> gym</span></code></pre></div>
<div class="sourceCode" id="cb7"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a>env <span class="op">=</span> gym.make(<span class="st">&#39;MountainCarContinuous-v0&#39;</span>)</span></code></pre></div>
<p>Our goal in this section is to find the parameters <span class="math inline">\(\theta\)</span> of the linear controller such that</p>
<p><span class="math display">\[
\theta^* = arg \max_{\theta} R_T(\theta).
\]</span></p>
<p>In this section, we directly use Bayesian optimization to solve this problem. We will use <a href="https://emukit.github.io">EmuKit</a> so we first define the objective function.</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> mountain_car <span class="im">as</span> mc</span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span></code></pre></div>
<p>For each set of parameter values of the linear controller we can run an episode of the simulator (that we fix to have a horizon of <span class="math inline">\(T=500\)</span>) to generate the reward. Using as input the parameters of the controller and as outputs the rewards we can build a Gaussian process emulator of the reward.</p>
<p>We start defining the input space, which is three-dimensional:</p>
<div class="sourceCode" id="cb9"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb9-1"><a href="#cb9-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core <span class="im">import</span> ContinuousParameter, ParameterSpace</span></code></pre></div>
<div class="sourceCode" id="cb10"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb10-1"><a href="#cb10-1" aria-hidden="true" tabindex="-1"></a>position_domain <span class="op">=</span> [<span class="op">-</span><span class="fl">1.2</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb10-2"><a href="#cb10-2" aria-hidden="true" tabindex="-1"></a>velocity_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span><span class="op">/</span><span class="fl">0.07</span>, <span class="op">+</span><span class="dv">1</span><span class="op">/</span><span class="fl">0.07</span>]</span>
<span id="cb10-3"><a href="#cb10-3" aria-hidden="true" tabindex="-1"></a>constant_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb10-4"><a href="#cb10-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb10-5"><a href="#cb10-5" aria-hidden="true" tabindex="-1"></a>space <span class="op">=</span> ParameterSpace(</span>
<span id="cb10-6"><a href="#cb10-6" aria-hidden="true" tabindex="-1"></a>          [ContinuousParameter(<span class="st">&#39;position_parameter&#39;</span>, <span class="op">*</span>position_domain), </span>
<span id="cb10-7"><a href="#cb10-7" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;velocity_parameter&#39;</span>, <span class="op">*</span>velocity_domain),</span>
<span id="cb10-8"><a href="#cb10-8" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;constant&#39;</span>, <span class="op">*</span>constant_domain)])</span></code></pre></div>
<p>To initalize the model we start sampling some initial points for the linear controller randomly.</p>
<div class="sourceCode" id="cb11"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core.initial_designs <span class="im">import</span> RandomDesign</span></code></pre></div>
<div class="sourceCode" id="cb12"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true" tabindex="-1"></a>design <span class="op">=</span> RandomDesign(space)</span>
<span id="cb12-2"><a href="#cb12-2" aria-hidden="true" tabindex="-1"></a>n_initial_points <span class="op">=</span> <span class="dv">25</span></span>
<span id="cb12-3"><a href="#cb12-3" aria-hidden="true" tabindex="-1"></a>initial_design <span class="op">=</span> design.get_samples(n_initial_points)</span></code></pre></div>
<p>Now run the simulation 25 times across our initial design.</p>
<div class="sourceCode" id="cb13"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a>y <span class="op">=</span> target_function(initial_design)</span></code></pre></div>
<p>Before we start any optimization, lets have a look to the behaviour of the car with the first of these initial points that we have selected randomly.</p>
<div class="sourceCode" id="cb14"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span></code></pre></div>
<p>This won’t render in Google <code>colab</code>, but should work in a regular Jupyter notebook if <code>pyglet</code> is installed. Details on rendering in <code>colab</code> are given in answer to this stackoverflow question <a href="https://stackoverflow.com/questions/50107530/how-to-render-openai-gym-in-google-colab" class="uri">https://stackoverflow.com/questions/50107530/how-to-render-openai-gym-in-google-colab</a>.</p>
<div class="sourceCode" id="cb15"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb15-1"><a href="#cb15-1" aria-hidden="true" tabindex="-1"></a>random_controller <span class="op">=</span> initial_design[<span class="dv">0</span>,:]</span>
<span id="cb15-2"><a href="#cb15-2" aria-hidden="true" tabindex="-1"></a>_, _, _, frames <span class="op">=</span> mc.run_simulation(env, np.atleast_2d(random_controller), render<span class="op">=</span><span class="va">True</span>)</span>
<span id="cb15-3"><a href="#cb15-3" aria-hidden="true" tabindex="-1"></a>anim<span class="op">=</span>mc.animate_frames(frames, <span class="st">&#39;Random linear controller&#39;</span>)</span></code></pre></div>
<div class="figure">
<div id="mountain-car-random-figure" class="figure-frame">
<iframe src="https://inverseprobability.com/slides/diagrams//uq/mountain-car-random.html" width="600" height="450" allowtransparency="true" frameborder="0">
</iframe>
</div>
<div id="mountain-car-random-magnify" class="magnify" onclick="magnifyFigure(&#39;mountain-car-random&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="mountain-car-random-caption" class="caption-frame">
<p>Figure: Random linear controller for the Mountain car. It fails to move the car to the top of the mountain.</p>
</div>
</div>
<p>As we can see the random linear controller does not manage to push the car to the top of the mountain. Now, let’s optimize the regret using Bayesian optimization and the emulator for the reward. We try 50 new parameters chosen by the expected improvement acquisition function.</p>
<p>First, we initizialize a Gaussian process emulator.</p>
<div class="sourceCode" id="cb16"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb16-1"><a href="#cb16-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> GPy</span></code></pre></div>
<div class="sourceCode" id="cb17"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb17-1"><a href="#cb17-1" aria-hidden="true" tabindex="-1"></a>kern <span class="op">=</span> GPy.kern.RBF(<span class="dv">3</span>)</span>
<span id="cb17-2"><a href="#cb17-2" aria-hidden="true" tabindex="-1"></a>model_gpy <span class="op">=</span> GPy.models.GPRegression(initial_design, y, kern, noise_var<span class="op">=</span><span class="fl">1e-10</span>)</span></code></pre></div>
<div class="sourceCode" id="cb18"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb18-1"><a href="#cb18-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.model_wrappers.gpy_model_wrappers <span class="im">import</span> GPyModelWrapper</span></code></pre></div>
<div class="sourceCode" id="cb19"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb19-1"><a href="#cb19-1" aria-hidden="true" tabindex="-1"></a>model_emukit <span class="op">=</span> GPyModelWrapper(model_gpy, n_restarts<span class="op">=</span><span class="dv">5</span>)</span>
<span id="cb19-2"><a href="#cb19-2" aria-hidden="true" tabindex="-1"></a>model_emukit.optimize()</span></code></pre></div>
<p>In Bayesian optimization an acquisition function is used to balance exploration and exploitation to evaluate new locations close to the optimum of the objective. In this notebook we select the expected improvement (EI). For further details have a look at the review paper of <span class="citation" data-cites="Shahriari-human16">Shahriari et al. (2016)</span>.</p>
<div class="sourceCode" id="cb20"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb20-1"><a href="#cb20-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.bayesian_optimization.acquisitions <span class="im">import</span> ExpectedImprovement</span></code></pre></div>
<div class="sourceCode" id="cb21"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb21-1"><a href="#cb21-1" aria-hidden="true" tabindex="-1"></a>acquisition <span class="op">=</span> ExpectedImprovement(model_emukit)</span></code></pre></div>
<div class="sourceCode" id="cb22"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb22-1"><a href="#cb22-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.bayesian_optimization.loops.bayesian_optimization_loop <span class="im">import</span> BayesianOptimizationLoop</span></code></pre></div>
<div class="sourceCode" id="cb23"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb23-1"><a href="#cb23-1" aria-hidden="true" tabindex="-1"></a>bo <span class="op">=</span> BayesianOptimizationLoop(space, model_emukit, acquisition<span class="op">=</span>acquisition)</span>
<span id="cb23-2"><a href="#cb23-2" aria-hidden="true" tabindex="-1"></a>bo.run_loop(target_function, <span class="dv">50</span>)</span>
<span id="cb23-3"><a href="#cb23-3" aria-hidden="true" tabindex="-1"></a>results<span class="op">=</span> bo.get_results()</span></code></pre></div>
<p>Now we visualize the result for the best controller that we have found with Bayesian optimization.</p>
<div class="sourceCode" id="cb24"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb24-1"><a href="#cb24-1" aria-hidden="true" tabindex="-1"></a>_, _, _, frames <span class="op">=</span> mc.run_simulation(env, np.atleast_2d(results.minimum_location), render<span class="op">=</span><span class="va">True</span>)</span>
<span id="cb24-2"><a href="#cb24-2" aria-hidden="true" tabindex="-1"></a>anim<span class="op">=</span>mc.animate_frames(frames, <span class="st">&#39;Best controller after 50 iterations of Bayesian optimization&#39;</span>)</span></code></pre></div>
<div class="figure">
<div id="mountain-car-similated-bayes-opt-figure" class="figure-frame">
<iframe src="https://inverseprobability.com/slides/diagrams//uq/mountain-car-simulated.html" width="600" height="450" allowtransparency="true" frameborder="0">
</iframe>
</div>
<div id="mountain-car-similated-bayes-opt-magnify" class="magnify" onclick="magnifyFigure(&#39;mountain-car-similated-bayes-opt&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="mountain-car-similated-bayes-opt-caption" class="caption-frame">
<p>Figure: Mountain car simulator trained using Bayesian optimization and the simulator of the dynamics. Fifty iterations of Bayesian optimization are used to optimize the controler.</p>
</div>
</div>
<p>The car can now make it to the top of the mountain! Emulating the reward function and using expected improvement acquisition helped us to find a linear controller that solves the problem.</p>
<h2 id="data-efficient-emulation">Data Efficient Emulation</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-data-efficient.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-data-efficient.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>In the previous section we solved the mountain car problem by directly emulating the reward but no considerations about the dynamics <span class="math display">\[
\mathbf{ x}_{t+1} =g(\mathbf{ x}_{t},\textbf{u}_{t})
\]</span> of the system were made.</p>
<p>We ran the simulator 25 times in the initial design, and 50 times in our Bayesian optimization loop. That required us to call the dynamics simulation <span class="math inline">\(500\times 75 =37,500\)</span> times, because each simulation of the car used 500 steps. In this section we will show how it is possible to reduce this number by building an emulator for <span class="math inline">\(g(\cdot)\)</span> that can later be used to directly optimize the control.</p>
<p>The inputs of the model for the dynamics are the velocity, the position and the value of the control so create this space accordingly.</p>
<div class="sourceCode" id="cb25"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb25-1"><a href="#cb25-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> gym</span></code></pre></div>
<div class="sourceCode" id="cb26"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb26-1"><a href="#cb26-1" aria-hidden="true" tabindex="-1"></a>env <span class="op">=</span> gym.make(<span class="st">&#39;MountainCarContinuous-v0&#39;</span>)</span></code></pre></div>
<div class="sourceCode" id="cb27"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb27-1"><a href="#cb27-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core <span class="im">import</span> ContinuousParameter, ParameterSpace</span></code></pre></div>
<div class="sourceCode" id="cb28"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb28-1"><a href="#cb28-1" aria-hidden="true" tabindex="-1"></a>position_dynamics_domain <span class="op">=</span> [<span class="op">-</span><span class="fl">1.2</span>, <span class="op">+</span><span class="fl">0.6</span>]</span>
<span id="cb28-2"><a href="#cb28-2" aria-hidden="true" tabindex="-1"></a>velocity_dynamics_domain <span class="op">=</span> [<span class="op">-</span><span class="fl">0.07</span>, <span class="op">+</span><span class="fl">0.07</span>]</span>
<span id="cb28-3"><a href="#cb28-3" aria-hidden="true" tabindex="-1"></a>action_dynamics_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb28-4"><a href="#cb28-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb28-5"><a href="#cb28-5" aria-hidden="true" tabindex="-1"></a>space_dynamics <span class="op">=</span> ParameterSpace(</span>
<span id="cb28-6"><a href="#cb28-6" aria-hidden="true" tabindex="-1"></a>          [ContinuousParameter(<span class="st">&#39;position_dynamics_parameter&#39;</span>, <span class="op">*</span>position_dynamics_domain), </span>
<span id="cb28-7"><a href="#cb28-7" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;velocity_dynamics_parameter&#39;</span>, <span class="op">*</span>velocity_dynamics_domain),</span>
<span id="cb28-8"><a href="#cb28-8" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;action_dynamics_parameter&#39;</span>, <span class="op">*</span>action_dynamics_domain)])</span></code></pre></div>
<p>Next, we sample some input parameters and use the simulator to compute the outputs. Note that in this case we are not running the full episodes, we are just using the simulator to compute <span class="math inline">\(\mathbf{ x}_{t+1}\)</span> given <span class="math inline">\(\mathbf{ x}_{t}\)</span> and <span class="math inline">\(\textbf{u}_{t}\)</span>.</p>
<div class="sourceCode" id="cb29"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb29-1"><a href="#cb29-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core.initial_designs <span class="im">import</span> RandomDesign</span></code></pre></div>
<div class="sourceCode" id="cb30"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb30-1"><a href="#cb30-1" aria-hidden="true" tabindex="-1"></a>design_dynamics <span class="op">=</span> RandomDesign(space_dynamics)</span>
<span id="cb30-2"><a href="#cb30-2" aria-hidden="true" tabindex="-1"></a>n_initial_points <span class="op">=</span> <span class="dv">500</span></span>
<span id="cb30-3"><a href="#cb30-3" aria-hidden="true" tabindex="-1"></a>initial_design_dynamics <span class="op">=</span> design_dynamics.get_samples(n_initial_points)</span></code></pre></div>
<div class="sourceCode" id="cb31"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb31-1"><a href="#cb31-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb31-2"><a href="#cb31-2" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> mountain_car <span class="im">as</span> mc</span></code></pre></div>
<div class="sourceCode" id="cb32"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb32-1"><a href="#cb32-1" aria-hidden="true" tabindex="-1"></a><span class="co">### --- Simulation of the (normalized) outputs</span></span>
<span id="cb32-2"><a href="#cb32-2" aria-hidden="true" tabindex="-1"></a>y_dynamics <span class="op">=</span> np.zeros((initial_design_dynamics.shape[<span class="dv">0</span>], <span class="dv">2</span>))</span>
<span id="cb32-3"><a href="#cb32-3" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(initial_design_dynamics.shape[<span class="dv">0</span>]):</span>
<span id="cb32-4"><a href="#cb32-4" aria-hidden="true" tabindex="-1"></a>    y_dynamics[i, :] <span class="op">=</span> mc.simulation(initial_design_dynamics[i, :])</span></code></pre></div>
<div class="sourceCode" id="cb33"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb33-1"><a href="#cb33-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Normalize the data from the simulation</span></span>
<span id="cb33-2"><a href="#cb33-2" aria-hidden="true" tabindex="-1"></a>y_dynamics_normalisation <span class="op">=</span> np.std(y_dynamics, axis<span class="op">=</span><span class="dv">0</span>)</span>
<span id="cb33-3"><a href="#cb33-3" aria-hidden="true" tabindex="-1"></a>y_dynamics_normalised <span class="op">=</span> y_dynamics<span class="op">/</span>y_dynamics_normalisation</span></code></pre></div>
<p>The outputs are the velocity and the position. Our model will capture the change in position and velocity on time. That is, we will model</p>
<p><span class="math display">\[
\Delta v_{t+1} = v_{t+1} - v_{t}
\]</span></p>
<p><span class="math display">\[
\Delta x_{t+1} = p_{t+1} - p_{t}
\]</span></p>
<p>with Gaussian processes with prior mean <span class="math inline">\(v_{t}\)</span> and <span class="math inline">\(p_{t}\)</span> respectively. As a covariance function, we use <code>Matern52</code>. We need therefore two models to capture the full dynamics of the system.</p>
<div class="sourceCode" id="cb34"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb34-1"><a href="#cb34-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> GPy</span></code></pre></div>
<div class="sourceCode" id="cb35"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb35-1"><a href="#cb35-1" aria-hidden="true" tabindex="-1"></a>kern_position <span class="op">=</span> GPy.kern.Matern52(<span class="dv">3</span>)</span>
<span id="cb35-2"><a href="#cb35-2" aria-hidden="true" tabindex="-1"></a>position_model_gpy <span class="op">=</span> GPy.models.GPRegression(initial_design_dynamics, y_dynamics[:, <span class="dv">0</span>:<span class="dv">1</span>], kern_position, noise_var<span class="op">=</span><span class="fl">1e-10</span>)</span></code></pre></div>
<div class="sourceCode" id="cb36"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb36-1"><a href="#cb36-1" aria-hidden="true" tabindex="-1"></a>kern_velocity <span class="op">=</span> GPy.kern.Matern52(<span class="dv">3</span>)</span>
<span id="cb36-2"><a href="#cb36-2" aria-hidden="true" tabindex="-1"></a>velocity_model_gpy <span class="op">=</span> GPy.models.GPRegression(initial_design_dynamics, y_dynamics[:, <span class="dv">1</span>:<span class="dv">2</span>], kern_velocity, noise_var<span class="op">=</span><span class="fl">1e-10</span>)</span></code></pre></div>
<div class="sourceCode" id="cb37"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb37-1"><a href="#cb37-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.model_wrappers.gpy_model_wrappers <span class="im">import</span> GPyModelWrapper</span></code></pre></div>
<div class="sourceCode" id="cb38"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb38-1"><a href="#cb38-1" aria-hidden="true" tabindex="-1"></a>position_model_emukit <span class="op">=</span> GPyModelWrapper(position_model_gpy, n_restarts<span class="op">=</span><span class="dv">5</span>)</span>
<span id="cb38-2"><a href="#cb38-2" aria-hidden="true" tabindex="-1"></a>velocity_model_emukit <span class="op">=</span> GPyModelWrapper(velocity_model_gpy, n_restarts<span class="op">=</span><span class="dv">5</span>)</span></code></pre></div>
<p>In general, we might use much smarter strategies to design our emulation of the simulator. For example, we could use the variance of the predictive distributions of the models to collect points using uncertainty sampling, which will give us a better coverage of the space. For simplicity, we move ahead with the 500 randomly selected points.</p>
<p>Now that we have a data set, we can update the emulators for the location and the velocity.</p>
<div class="sourceCode" id="cb39"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb39-1"><a href="#cb39-1" aria-hidden="true" tabindex="-1"></a>position_model_emukit.optimize()</span>
<span id="cb39-2"><a href="#cb39-2" aria-hidden="true" tabindex="-1"></a>velocity_model_emukit.optimize()</span></code></pre></div>
<p>We can now have a look to how the emulator and the simulator match. First, we show a contour plot of the car acceleration for each pair of can position and velocity. You can use the bar bellow to play with the values of the controller to compare the emulator and the simulator.</p>
<p>We can see how the emulator is doing a fairly good job approximating the simulator. On the edges, however, it struggles to captures the dynamics of the simulator.</p>
<p>Given some input parameters of the linear controlling, how do the dynamics of the emulator and simulator match? In the following figure we show the position and velocity of the car for the 500 time-steps of an episode in which the parameters of the linear controller have been fixed beforehand. The value of the input control is also shown.</p>
<div class="sourceCode" id="cb40"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb40-1"><a href="#cb40-1" aria-hidden="true" tabindex="-1"></a><span class="co"># change the values of the linear controller to observe the trajectories.</span></span>
<span id="cb40-2"><a href="#cb40-2" aria-hidden="true" tabindex="-1"></a>controller_gains <span class="op">=</span> np.atleast_2d([<span class="dv">0</span>, <span class="fl">.6</span>, <span class="dv">1</span>])  </span></code></pre></div>
<div class="figure">
<div id="emu-sim-comparison-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//uq/emu-sim-comparison.svg" width="80%" style=" ">
</object>
</div>
<div id="emu-sim-comparison-magnify" class="magnify" onclick="magnifyFigure(&#39;emu-sim-comparison&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="emu-sim-comparison-caption" class="caption-frame">
<p>Figure: Comparison between the mountain car simulator and the emulator.</p>
</div>
</div>
<p>We now make explicit use of the emulator, using it to replace the simulator and optimize the linear controller. Note that in this optimization, we don’t need to query the simulator anymore as we can reproduce the full dynamics of an episode using the emulator. For illustrative purposes, in this example we fix the initial location of the car.</p>
<p>We define the objective reward function in terms of the simulator.</p>
<div class="sourceCode" id="cb41"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb41-1"><a href="#cb41-1" aria-hidden="true" tabindex="-1"></a><span class="co">### --- Optimize control parameters with emulator</span></span>
<span id="cb41-2"><a href="#cb41-2" aria-hidden="true" tabindex="-1"></a>car_initial_location <span class="op">=</span> np.asarray([<span class="op">-</span><span class="fl">0.58912799</span>, <span class="dv">0</span>])</span></code></pre></div>
<p>And as before, we use Bayesian optimization to find the best possible linear controller.</p>
<p>The design space is the three continuous variables that make up the linear controller.</p>
<div class="sourceCode" id="cb42"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb42-1"><a href="#cb42-1" aria-hidden="true" tabindex="-1"></a>position_domain <span class="op">=</span> [<span class="op">-</span><span class="fl">1.2</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb42-2"><a href="#cb42-2" aria-hidden="true" tabindex="-1"></a>velocity_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span><span class="op">/</span><span class="fl">0.07</span>, <span class="op">+</span><span class="dv">1</span><span class="op">/</span><span class="fl">0.07</span>]</span>
<span id="cb42-3"><a href="#cb42-3" aria-hidden="true" tabindex="-1"></a>constant_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb42-4"><a href="#cb42-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb42-5"><a href="#cb42-5" aria-hidden="true" tabindex="-1"></a>space <span class="op">=</span> ParameterSpace(</span>
<span id="cb42-6"><a href="#cb42-6" aria-hidden="true" tabindex="-1"></a>          [ContinuousParameter(<span class="st">&#39;position_parameter&#39;</span>, <span class="op">*</span>position_domain), </span>
<span id="cb42-7"><a href="#cb42-7" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;velocity_parameter&#39;</span>, <span class="op">*</span>velocity_domain),</span>
<span id="cb42-8"><a href="#cb42-8" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;constant&#39;</span>, <span class="op">*</span>constant_domain)])</span></code></pre></div>
<!--```{.python}

```{space= [{'name':'linear_1', 'type':'continuous', 'domain':(-1/1.2, +1)},
        {'name':'linear_2', 'type':'continuous', 'domain':(-1/0.07, +1/0.07)},
        {'name':'constant', 'type':'continuous', 'domain':(-1, +1)}]-->
<div class="sourceCode" id="cb43"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb43-1"><a href="#cb43-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core.initial_designs <span class="im">import</span> RandomDesign</span></code></pre></div>
<div class="sourceCode" id="cb44"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb44-1"><a href="#cb44-1" aria-hidden="true" tabindex="-1"></a>design <span class="op">=</span> RandomDesign(space)</span>
<span id="cb44-2"><a href="#cb44-2" aria-hidden="true" tabindex="-1"></a>n_initial_points <span class="op">=</span> <span class="dv">25</span></span>
<span id="cb44-3"><a href="#cb44-3" aria-hidden="true" tabindex="-1"></a>initial_design <span class="op">=</span> design.get_samples(n_initial_points)</span></code></pre></div>
<p>Now run the simulation 25 times across our initial design.</p>
<div class="sourceCode" id="cb45"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb45-1"><a href="#cb45-1" aria-hidden="true" tabindex="-1"></a>y <span class="op">=</span> target_function_emulator(initial_design)</span></code></pre></div>
<p>Now we set up the surrogate model for the Bayesian optimization loop.</p>
<div class="sourceCode" id="cb46"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb46-1"><a href="#cb46-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> GPy</span></code></pre></div>
<div class="sourceCode" id="cb47"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb47-1"><a href="#cb47-1" aria-hidden="true" tabindex="-1"></a>kern <span class="op">=</span> GPy.kern.RBF(<span class="dv">3</span>)</span>
<span id="cb47-2"><a href="#cb47-2" aria-hidden="true" tabindex="-1"></a>model_dynamics_emulated_gpy <span class="op">=</span> GPy.models.GPRegression(initial_design, y, kern, noise_var<span class="op">=</span><span class="fl">1e-10</span>)</span></code></pre></div>
<div class="sourceCode" id="cb48"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb48-1"><a href="#cb48-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.model_wrappers.gpy_model_wrappers <span class="im">import</span> GPyModelWrapper</span></code></pre></div>
<div class="sourceCode" id="cb49"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb49-1"><a href="#cb49-1" aria-hidden="true" tabindex="-1"></a>model_dynamics_emulated_emukit <span class="op">=</span> GPyModelWrapper(model_dynamics_emulated_gpy, n_restarts<span class="op">=</span><span class="dv">5</span>)</span>
<span id="cb49-2"><a href="#cb49-2" aria-hidden="true" tabindex="-1"></a>model_dynamics_emulated_emukit.optimize()</span></code></pre></div>
<p>We set the acquisition function to be expected improvement.</p>
<div class="sourceCode" id="cb50"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb50-1"><a href="#cb50-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.bayesian_optimization.acquisitions <span class="im">import</span> ExpectedImprovement</span></code></pre></div>
<div class="sourceCode" id="cb51"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb51-1"><a href="#cb51-1" aria-hidden="true" tabindex="-1"></a>acquisition <span class="op">=</span> ExpectedImprovement(model_emukit)</span></code></pre></div>
<p>And we set up the main loop for the Bayesian optimization.</p>
<div class="sourceCode" id="cb52"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb52-1"><a href="#cb52-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.bayesian_optimization.loops.bayesian_optimization_loop <span class="im">import</span> BayesianOptimizationLoop</span></code></pre></div>
<div class="sourceCode" id="cb53"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb53-1"><a href="#cb53-1" aria-hidden="true" tabindex="-1"></a>bo <span class="op">=</span> BayesianOptimizationLoop(space, model_dynamics_emulated_emukit, acquisition<span class="op">=</span>acquisition)</span>
<span id="cb53-2"><a href="#cb53-2" aria-hidden="true" tabindex="-1"></a>bo.run_loop(target_function_emulator, <span class="dv">50</span>)</span>
<span id="cb53-3"><a href="#cb53-3" aria-hidden="true" tabindex="-1"></a>results <span class="op">=</span> bo.get_results()</span></code></pre></div>
<div class="sourceCode" id="cb54"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb54-1"><a href="#cb54-1" aria-hidden="true" tabindex="-1"></a>_, _, _, frames <span class="op">=</span> mc.run_simulation(env, np.atleast_2d(results.minimum_location), render<span class="op">=</span><span class="va">True</span>)</span>
<span id="cb54-2"><a href="#cb54-2" aria-hidden="true" tabindex="-1"></a>anim<span class="op">=</span>mc.animate_frames(frames, <span class="st">&#39;Best controller using the emulator of the dynamics&#39;</span>)</span></code></pre></div>
<div class="sourceCode" id="cb55"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb55-1"><a href="#cb55-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> IPython.core.display <span class="im">import</span> HTML</span></code></pre></div>
<div class="figure">
<div id="mountain-car-emulated-figure" class="figure-frame">
<iframe src="https://inverseprobability.com/slides/diagrams//uq/mountain-car-emulated.html" width="600" height="450" allowtransparency="true" frameborder="0">
</iframe>
</div>
<div id="mountain-car-emulated-magnify" class="magnify" onclick="magnifyFigure(&#39;mountain-car-emulated&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="mountain-car-emulated-caption" class="caption-frame">
<p>Figure: Mountain car controller learnt through emulation. Here 500 calls to the simulator are used to fit the controller rather than 37,500 calls to the simulator required in the standard learning.</p>
</div>
</div>
<p>And the problem is again solved, but in this case, we have replaced the simulator of the car dynamics by a Gaussian process emulator that we learned by calling the dynamics simulator only 500 times. Compared to the 37,500 calls that we needed when applying Bayesian optimization directly on the simulator this is a significant improvement. Of course, in practice the car dynamics are very simple for this example.</p>
<h2 id="mountain-car-multi-fidelity-emulation">Mountain Car: Multi-Fidelity Emulation</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-multi-fidelity-introduction.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-multi-fidelity-introduction.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>In some scenarios we have simulators of the same environment that have different fidelities, that is that reflect with different level of accuracy the dynamics of the real world. Running simulations of the different fidelities also have a different cost: high-fidelity simulations are typically more expensive the low-fidelity. If we have access to these simulators, we can combine high and low-fidelity simulations under the same model.</p>
<p>So, let’s assume that we have two simulators of the mountain car dynamics, one of high fidelity (the one we have used) and another one of low fidelity. The traditional approach to this form of multi-fidelity emulation is to assume that <span class="math display">\[
f_i\left(\mathbf{ x}\right) = \rho f_{i-1}\left(\mathbf{ x}\right) + \delta_i\left(\mathbf{ x}\right),
\]</span> where <span class="math inline">\(f_{i-1}\left(\mathbf{ x}\right)\)</span> is a low-fidelity simulation of the problem of interest and <span class="math inline">\(f_i\left(\mathbf{ x}\right)\)</span> is a higher fidelity simulation. The function <span class="math inline">\(\delta_i\left(\mathbf{ x}\right)\)</span> represents the difference between the lower and higher fidelity simulation, which is considered additive. The additive form of this covariance means that if <span class="math inline">\(f_{0}\left(\mathbf{ x}\right)\)</span> and <span class="math inline">\(\left\{\delta_i\left(\mathbf{ x}\right)\right\}_{i=1}^m\)</span> are all Gaussian processes, then the process over all fidelities of simulation will be a joint Gaussian process.</p>
<p>But with deep Gaussian processes we can consider the form <span class="math display">\[
f_i\left(\mathbf{ x}\right) = g_{i}\left(f_{i-1}\left(\mathbf{ x}\right)\right) + \delta_i\left(\mathbf{ x}\right),
\]</span> where the low fidelity representation is nonlinearly transformed by <span class="math inline">\(g(\cdot)\)</span> before use in the process. This is the approach taken in <span class="citation" data-cites="Perdikaris:multifidelity17">Perdikaris et al. (2017)</span>. But once we accept that these models can be composed, a highly flexible framework can emerge. A key point is that the data enters the model at different levels and represents different aspects. For example, these correspond to the two fidelities of the mountain car simulator.</p>
<p>We start by sampling both at 250 random input locations.</p>
<div class="sourceCode" id="cb56"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb56-1"><a href="#cb56-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> gym</span></code></pre></div>
<div class="sourceCode" id="cb57"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb57-1"><a href="#cb57-1" aria-hidden="true" tabindex="-1"></a>env <span class="op">=</span> gym.make(<span class="st">&#39;MountainCarContinuous-v0&#39;</span>)</span></code></pre></div>
<div class="sourceCode" id="cb58"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb58-1"><a href="#cb58-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core <span class="im">import</span> ContinuousParameter, ParameterSpace</span></code></pre></div>
<div class="sourceCode" id="cb59"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb59-1"><a href="#cb59-1" aria-hidden="true" tabindex="-1"></a>position_dynamics_domain <span class="op">=</span> [<span class="op">-</span><span class="fl">1.2</span>, <span class="op">+</span><span class="fl">0.6</span>]</span>
<span id="cb59-2"><a href="#cb59-2" aria-hidden="true" tabindex="-1"></a>velocity_dynamics_domain <span class="op">=</span> [<span class="op">-</span><span class="fl">0.07</span>, <span class="op">+</span><span class="fl">0.07</span>]</span>
<span id="cb59-3"><a href="#cb59-3" aria-hidden="true" tabindex="-1"></a>action_dynamics_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb59-4"><a href="#cb59-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb59-5"><a href="#cb59-5" aria-hidden="true" tabindex="-1"></a>space_dynamics <span class="op">=</span> ParameterSpace(</span>
<span id="cb59-6"><a href="#cb59-6" aria-hidden="true" tabindex="-1"></a>          [ContinuousParameter(<span class="st">&#39;position_dynamics_parameter&#39;</span>, <span class="op">*</span>position_dynamics_domain), </span>
<span id="cb59-7"><a href="#cb59-7" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;velocity_dynamics_parameter&#39;</span>, <span class="op">*</span>velocity_dynamics_domain),</span>
<span id="cb59-8"><a href="#cb59-8" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;action_dynamics_parameter&#39;</span>, <span class="op">*</span>action_dynamics_domain)])</span></code></pre></div>
<p>Next, we evaluate the high and low fidelity simualtors at those locations.</p>
<div class="sourceCode" id="cb60"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb60-1"><a href="#cb60-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb60-2"><a href="#cb60-2" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> mountain_car <span class="im">as</span> mc</span></code></pre></div>
<div class="sourceCode" id="cb61"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb61-1"><a href="#cb61-1" aria-hidden="true" tabindex="-1"></a>n_points <span class="op">=</span> <span class="dv">250</span></span>
<span id="cb61-2"><a href="#cb61-2" aria-hidden="true" tabindex="-1"></a>d_position_hf <span class="op">=</span> np.zeros((n_points, <span class="dv">1</span>))</span>
<span id="cb61-3"><a href="#cb61-3" aria-hidden="true" tabindex="-1"></a>d_velocity_hf <span class="op">=</span> np.zeros((n_points, <span class="dv">1</span>))</span>
<span id="cb61-4"><a href="#cb61-4" aria-hidden="true" tabindex="-1"></a>d_position_lf <span class="op">=</span> np.zeros((n_points, <span class="dv">1</span>))</span>
<span id="cb61-5"><a href="#cb61-5" aria-hidden="true" tabindex="-1"></a>d_velocity_lf <span class="op">=</span> np.zeros((n_points, <span class="dv">1</span>))</span>
<span id="cb61-6"><a href="#cb61-6" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb61-7"><a href="#cb61-7" aria-hidden="true" tabindex="-1"></a><span class="co"># --- Collect high fidelity points</span></span>
<span id="cb61-8"><a href="#cb61-8" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, n_points):</span>
<span id="cb61-9"><a href="#cb61-9" aria-hidden="true" tabindex="-1"></a>    d_position_hf[i], d_velocity_hf[i] <span class="op">=</span> mc.simulation(x_random[i, :])</span>
<span id="cb61-10"><a href="#cb61-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb61-11"><a href="#cb61-11" aria-hidden="true" tabindex="-1"></a><span class="co"># --- Collect low fidelity points  </span></span>
<span id="cb61-12"><a href="#cb61-12" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> i <span class="kw">in</span> <span class="bu">range</span>(<span class="dv">0</span>, n_points):</span>
<span id="cb61-13"><a href="#cb61-13" aria-hidden="true" tabindex="-1"></a>    d_position_lf[i], d_velocity_lf[i] <span class="op">=</span> mc.low_cost_simulation(x_random[i, :])</span></code></pre></div>
<h2 id="building-the-multifidelity-emulation">Building the Multifidelity Emulation</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-multi-fidelity-solution.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/mountain-car-multi-fidelity-solution.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>It is time to build the multi-fidelity model for both the position and the velocity.</p>
<p>As we did in the previous section we use the emulator to optimize the simulator. In this case we use the high fidelity output of the emulator.</p>
<p>First we optimize the controller parameters</p>
<!--code{obj_func = lambda x: mc.run_simulation(env, x)[0]
obj_func_emulator = lambda x: mc.run_emulation([position_model, velocity_model], x, car_initial_location)[0]
objective_multifidelity = GPyOpt.core.task.SingleObjective(obj_func)}-->
<p>And we optimize using Bayesian optimzation.</p>
<div class="sourceCode" id="cb62"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb62-1"><a href="#cb62-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core <span class="im">import</span> ContinuousParameter, ParameterSpace</span></code></pre></div>
<div class="sourceCode" id="cb63"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb63-1"><a href="#cb63-1" aria-hidden="true" tabindex="-1"></a>position_domain <span class="op">=</span> [<span class="op">-</span><span class="fl">1.2</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb63-2"><a href="#cb63-2" aria-hidden="true" tabindex="-1"></a>velocity_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span><span class="op">/</span><span class="fl">0.07</span>, <span class="op">+</span><span class="dv">1</span><span class="op">/</span><span class="fl">0.07</span>]</span>
<span id="cb63-3"><a href="#cb63-3" aria-hidden="true" tabindex="-1"></a>constant_domain <span class="op">=</span> [<span class="op">-</span><span class="dv">1</span>, <span class="op">+</span><span class="dv">1</span>]</span>
<span id="cb63-4"><a href="#cb63-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb63-5"><a href="#cb63-5" aria-hidden="true" tabindex="-1"></a>space <span class="op">=</span> ParameterSpace(</span>
<span id="cb63-6"><a href="#cb63-6" aria-hidden="true" tabindex="-1"></a>          [ContinuousParameter(<span class="st">&#39;position_parameter&#39;</span>, <span class="op">*</span>position_domain), </span>
<span id="cb63-7"><a href="#cb63-7" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;velocity_parameter&#39;</span>, <span class="op">*</span>velocity_domain),</span>
<span id="cb63-8"><a href="#cb63-8" aria-hidden="true" tabindex="-1"></a>           ContinuousParameter(<span class="st">&#39;constant&#39;</span>, <span class="op">*</span>constant_domain)])</span></code></pre></div>
<div class="sourceCode" id="cb64"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb64-1"><a href="#cb64-1" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> emukit.core.initial_designs <span class="im">import</span> RandomDesign</span></code></pre></div>
<div class="sourceCode" id="cb65"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb65-1"><a href="#cb65-1" aria-hidden="true" tabindex="-1"></a>design <span class="op">=</span> RandomDesign(space)</span>
<span id="cb65-2"><a href="#cb65-2" aria-hidden="true" tabindex="-1"></a>n_initial_points <span class="op">=</span> <span class="dv">25</span></span>
<span id="cb65-3"><a href="#cb65-3" aria-hidden="true" tabindex="-1"></a>initial_design <span class="op">=</span> design.get_samples(n_initial_points)</span></code></pre></div>
<p>n_initial_points = 25 random_design = RandomDesign(design_space) initial_design = random_design.get_samples(n_initial_points) acquisition = GPyOpt.acquisitions.AcquisitionEI(model, design_space, optimizer=aquisition_optimizer) evaluator = GPyOpt.core.evaluators.Sequential(acquisition)}</p>
<div class="sourceCode" id="cb66"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb66-1"><a href="#cb66-1" aria-hidden="true" tabindex="-1"></a>bo_multifidelity <span class="op">=</span> GPyOpt.methods.ModularBayesianOptimization(model, design_space, objective_multifidelity, acquisition, evaluator, initial_design)</span>
<span id="cb66-2"><a href="#cb66-2" aria-hidden="true" tabindex="-1"></a>bo_multifidelity.run_optimization(max_iter<span class="op">=</span><span class="dv">50</span>)</span></code></pre></div>
<div class="sourceCode" id="cb67"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb67-1"><a href="#cb67-1" aria-hidden="true" tabindex="-1"></a>_, _, _, frames <span class="op">=</span> mc.run_simulation(env, np.atleast_2d(bo_multifidelity.x_opt), render<span class="op">=</span><span class="va">True</span>)</span>
<span id="cb67-2"><a href="#cb67-2" aria-hidden="true" tabindex="-1"></a>anim<span class="op">=</span>mc.animate_frames(frames, <span class="st">&#39;Best controller with multi-fidelity emulator&#39;</span>)</span></code></pre></div>
<h3 id="best-controller-with-multi-fidelity-emulator">Best Controller with Multi-Fidelity Emulator</h3>
<div class="figure">
<div id="mountain-car-multi-fidelity-figure" class="figure-frame">
<iframe src="https://inverseprobability.com/slides/diagrams//uq/mountain-car-multi-fidelity.html" width="800px" height="600px" allowtransparency="true" frameborder="0">
</iframe>
</div>
<div id="mountain-car-multi-fidelity-magnify" class="magnify" onclick="magnifyFigure(&#39;mountain-car-multi-fidelity&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="mountain-car-multi-fidelity-caption" class="caption-frame">
<p>Figure: Mountain car learnt with multi-fidelity model. Here 250 observations of the high fidelity simulator and 250 observations of the low fidelity simulator are used to learn the controller.</p>
</div>
</div>
<p>And problem solved! We see how the problem is also solved with 250 observations of the high fidelity simulator and 250 of the low fidelity simulator.</p>
<h2 id="emukit-playground">Emukit Playground</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_uq/includes/emukit-playground.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_uq/includes/emukit-playground.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<svg viewBox="0 0 200 200" style="width:15%">
<defs> <clipPath id="clip0">
<style>
circle {
  fill: black;
}
</style>
<circle cx="100" cy="100" r="100"/> </clipPath> </defs>
<title>
Leah Hirst
</title>
<image preserveAspectRatio="xMinYMin slice" width="100%" xlink:href="https://inverseprobability.com/slides/diagrams//people/person-placeholder.jpg" clip-path="url(#clip0)"/>
</svg>
<svg viewBox="0 0 200 200" style="width:15%">
<defs> <clipPath id="clip1">
<style>
circle {
  fill: black;
}
</style>
<circle cx="100" cy="100" r="100"/> </clipPath> </defs>
<title>
Cliff McCollum
</title>
<image preserveAspectRatio="xMinYMin slice" width="100%" xlink:href="https://inverseprobability.com/slides/diagrams//people/cliff-mccollum.jpg" clip-path="url(#clip1)"/>
</svg>
<p>Emukit playground is a software toolkit for exploring the use of statistical emulation as a tool. It was built by <a href="https://www.linkedin.com/in/leahhirst/">Leah Hirst</a>, during her software engineering internship at Amazon and supervised by <a href="https://www.linkedin.com/in/cliffmccollum/">Cliff McCollum</a>.</p>
<div class="figure">
<div id="emukit-playground-figure" class="figure-frame">
<div class="centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//uq/emukit-playground.png" width="80%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="emukit-playground-magnify" class="magnify" onclick="magnifyFigure(&#39;emukit-playground&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="emukit-playground-caption" class="caption-frame">
<p>Figure: Emukit playground is a tutorial for understanding the simulation/emulation relationship. <a href="https://amzn.github.io/emukit-playground/" class="uri">https://amzn.github.io/emukit-playground/</a></p>
</div>
</div>
<div class="figure">
<div id="emukit-playground-bayes-opt-figure" class="figure-frame">
<div class="centered" style="">
<img class="negate" src="https://inverseprobability.com/slides/diagrams//uq/emukit-playground-bayes-opt.png" width="80%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="emukit-playground-bayes-opt-magnify" class="magnify" onclick="magnifyFigure(&#39;emukit-playground-bayes-opt&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="emukit-playground-bayes-opt-caption" class="caption-frame">
<p>Figure: Tutorial on Bayesian optimization of the number of taxis deployed from Emukit playground. <a href="https://amzn.github.io/emukit-playground/#!/learn/bayesian_optimization" class="uri">https://amzn.github.io/emukit-playground/#!/learn/bayesian_optimization</a></p>
</div>
</div>
<p>You can explore Bayesian optimization of a taxi simulation.</p>
<h1 id="emukit">Emukit</h1>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_software/includes/emukit-software.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_software/includes/emukit-software.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<svg viewBox="0 0 200 200" style="width:15%">
<defs> <clipPath id="clip2">
<style>
circle {
  fill: black;
}
</style>
<circle cx="100" cy="100" r="100"/> </clipPath> </defs>
<title>
Javier Gonzalez
</title>
<image preserveAspectRatio="xMinYMin slice" width="100%" xlink:href="https://inverseprobability.com/slides/diagrams//people/javier-gonzalez.png" clip-path="url(#clip2)"/>
</svg>
<p>The Emukit software we will be using across the next part of this module is a python software library that facilitates emulation of systems. The software’s origins go back to work done by <a href="https://javiergonzalezh.github.io/">Javier Gonzalez</a> as part of his post-doctoral project at the University of Sheffield. Javier led the design and build of a Bayesian optimization software. The package <code>GPyOpt</code> worked with the SheffieldML software GPy for performing Bayesian optimization.</p>
<p><code>GPyOpt</code> has a modular design that allows the user to provide their own surrogate models, the package is build with <code>GPy</code> as a surrogate model in mind, but other surrogate models can also be wrapped and integrated.</p>
<p>However, <code>GPyOpt</code> doesn’t allow the full flexibility of surrogate modelling for domains like experimental design, sensitivity analysis etc.</p>
<p>Emukit was designed and built for a more general approach. The software is MIT licensed and its design and implementation was led by Javier Gonzalez and <a href="https://www.linkedin.com/in/andreipaleyes">Andrei Paleyes</a> at Amazon. Building on the experience of <code>GPyOpt</code>, the aim with Emukit was to use the modularisation ideas embedded in <code>GPyOpt</code>, but to extend them beyond the modularisation of the surrogate models to modularisation of the acquisition function.</p>
<div class="figure">
<div id="emukit-software-page-figure" class="figure-frame">
<div class="centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//uq/emukit-software-page.png" width="80%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="emukit-software-page-magnify" class="magnify" onclick="magnifyFigure(&#39;emukit-software-page&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="emukit-software-page-caption" class="caption-frame">
<p>Figure: The Emukit software is a set of software tools for emulation and surrogate modeling. <a href="https://emukit.github.io/emukit/" class="uri">https://emukit.github.io/emukit/</a></p>
</div>
</div>
<div class="sourceCode" id="cb68"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb68-1"><a href="#cb68-1" aria-hidden="true" tabindex="-1"></a><span class="op">%</span>pip install gpy</span></code></pre></div>
<div class="sourceCode" id="cb69"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb69-1"><a href="#cb69-1" aria-hidden="true" tabindex="-1"></a><span class="op">%</span>pip install pyDOE</span></code></pre></div>
<div class="sourceCode" id="cb70"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb70-1"><a href="#cb70-1" aria-hidden="true" tabindex="-1"></a><span class="op">%</span>pip install emukit</span></code></pre></div>
<div class="centered" style="">
<svg viewBox="0 0 200 200" style="width:10%">
<defs> <clipPath id="clip3">
<style>
circle {
  fill: black;
}
</style>
<circle cx="100" cy="100" r="100"/> </clipPath> </defs>
<title>
Javier Gonzalez
</title>
<image preserveAspectRatio="xMinYMin slice" width="100%" xlink:href="https://inverseprobability.com/slides/diagrams//people/javier-gonzalez.png" clip-path="url(#clip3)"/>
</svg>
<svg viewBox="0 0 200 200" style="width:10%">
<defs> <clipPath id="clip4">
<style>
circle {
  fill: black;
}
</style>
<circle cx="100" cy="100" r="100"/> </clipPath> </defs>
<title>
Andrei Paleyes
</title>
<image preserveAspectRatio="xMinYMin slice" width="100%" xlink:href="https://inverseprobability.com/slides/diagrams//people/andrei-paleyes.jpg" clip-path="url(#clip4)"/>
</svg>
<svg viewBox="0 0 200 200" style="width:10%">
<defs> <clipPath id="clip5">
<style>
circle {
  fill: black;
}
</style>
<circle cx="100" cy="100" r="100"/> </clipPath> </defs>
<title>
Mark Pullin
</title>
<image preserveAspectRatio="xMinYMin slice" width="100%" xlink:href="https://inverseprobability.com/slides/diagrams//people/mark-pullin.jpg" clip-path="url(#clip5)"/>
</svg>
<svg viewBox="0 0 200 200" style="width:10%">
<defs> <clipPath id="clip6">
<style>
circle {
  fill: black;
}
</style>
<circle cx="100" cy="100" r="100"/> </clipPath> </defs>
<title>
Maren Mahsereci
</title>
<image preserveAspectRatio="xMinYMin slice" width="100%" xlink:href="https://inverseprobability.com/slides/diagrams//people/maren-mahsereci.png" clip-path="url(#clip6)"/>
</svg>
</div>
<p>The software was initially built by the team in Amazon. As well as Javier Gonzalez (ML side) and Andrei Paleyes (Software Engineering) included Mark Pullin, Maren Mahsereci, Alex Gessner, Aaron Klein, Henry Moss, David-Elias Künstle as well as management input from Cliff McCollum and myself.</p>
<p>{For monitoring systems in production, emulation needn’t just be about simulator models. What we envisage, is that even data driven models could be emulated. This is important for understanding system behaviour, how the different components are interconnected. This drives the notion of the <em>information dynamics</em> of the machine learning system. What is the effect of one particular intervention in the wider system? One way of answering this is through emulation. But it requires that our machine learning models (and our simulators) are deployed in an environment where emulation can be automatically deployed. The resulting system would allow us to monitor the downstream effects of indivdiual decision making on the wider system.</p>
<p>blog post on <a href="http://inverseprobability.com/2016/11/29/new-directions-in-kernels-and-gaussian-processes">New Directions in Kernels and Gaussian Processes</a>.</p>
<h1 id="deep-gaussian-processes">Deep Gaussian Processes</h1>
<p>One challenge is developing flexible enough models to perform the emulation that also propagate uncertainty through the model. One candidate set of models for this challenge is <em>deep Gaussian processes</em> (DGPs). For the remainder of these notes we introduce the theory behind DGPs.</p>
<p>While there are some difficulties in algorithmically implementing these algorithms at scale, they are mathematically far simpler than the equivalent neural network models, and perhaps as a result offer greater promise for theoretical understanding of deep learning <span class="citation" data-cites="Dunlop:deep2017">(see e.g. Dunlop et al., n.d.)</span>.</p>
<h2 id="bottleneck-layers-in-deep-neural-networks">Bottleneck Layers in Deep Neural Networks</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_deepgp/includes/deep-gp.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_deepgp/includes/deep-gp.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<div class="figure">
<div id="deep-nn-bottleneck-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//deepgp/deep-nn-bottleneck2.svg" width="70%" style=" ">
</object>
</div>
<div id="deep-nn-bottleneck-magnify" class="magnify" onclick="magnifyFigure(&#39;deep-nn-bottleneck&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="deep-nn-bottleneck-caption" class="caption-frame">
<p>Figure: Inserting the bottleneck layers introduces a new set of variables.</p>
</div>
</div>
<p>Including the low rank decomposition of <span class="math inline">\(\mathbf{W}\)</span> in the neural network, we obtain a new mathematical form. Effectively, we are adding additional <em>latent</em> layers, <span class="math inline">\(\mathbf{ z}\)</span>, in between each of the existing hidden layers. In a neural network these are sometimes known as <em>bottleneck</em> layers. The network can now be written mathematically as <span class="math display">\[
\begin{align}
  \mathbf{ z}_{1} &amp;= \mathbf{V}^\top_1 \mathbf{ x}\\
  \mathbf{ h}_{1} &amp;= \phi\left(\mathbf{U}_1 \mathbf{ z}_{1}\right)\\
  \mathbf{ z}_{2} &amp;= \mathbf{V}^\top_2 \mathbf{ h}_{1}\\
  \mathbf{ h}_{2} &amp;= \phi\left(\mathbf{U}_2 \mathbf{ z}_{2}\right)\\
  \mathbf{ z}_{3} &amp;= \mathbf{V}^\top_3 \mathbf{ h}_{2}\\
  \mathbf{ h}_{3} &amp;= \phi\left(\mathbf{U}_3 \mathbf{ z}_{3}\right)\\
  \mathbf{ y}&amp;= \mathbf{ w}_4^\top\mathbf{ h}_{3}.
\end{align}
\]</span></p>
<p><span class="math display">\[
\begin{align}
  \mathbf{ z}_{1} &amp;= \mathbf{V}^\top_1 \mathbf{ x}\\
  \mathbf{ z}_{2} &amp;= \mathbf{V}^\top_2 \phi\left(\mathbf{U}_1 \mathbf{ z}_{1}\right)\\
  \mathbf{ z}_{3} &amp;= \mathbf{V}^\top_3 \phi\left(\mathbf{U}_2 \mathbf{ z}_{2}\right)\\
  \mathbf{ y}&amp;= \mathbf{ w}_4 ^\top \mathbf{ z}_{3}
\end{align}
\]</span></p>
<h2 id="cascade-of-gaussian-processes">Cascade of Gaussian Processes</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_deepgp/includes/cascade-of-gps.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_deepgp/includes/cascade-of-gps.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>Now if we replace each of these neural networks with a Gaussian process. This is equivalent to taking the limit as the width of each layer goes to infinity, while appropriately scaling down the outputs.</p>
<p><span class="math display">\[
\begin{align}
  \mathbf{ z}_{1} &amp;= \mathbf{ f}_1\left(\mathbf{ x}\right)\\
  \mathbf{ z}_{2} &amp;= \mathbf{ f}_2\left(\mathbf{ z}_{1}\right)\\
  \mathbf{ z}_{3} &amp;= \mathbf{ f}_3\left(\mathbf{ z}_{2}\right)\\
  \mathbf{ y}&amp;= \mathbf{ f}_4\left(\mathbf{ z}_{3}\right)
\end{align}
\]</span></p>
<h2 id="olympic-marathon-data">Olympic Marathon Data</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_datasets/includes/olympic-marathon-data.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_datasets/includes/olympic-marathon-data.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<table>
<tr>
<td width="70%">
<ul>
<li>Gold medal times for Olympic Marathon since 1896.</li>
<li>Marathons before 1924 didn’t have a standardised distance.</li>
<li>Present results using pace per km.</li>
<li>In 1904 Marathon was badly organised leading to very slow times.</li>
</ul>
</td>
<td width="30%">
<div class="centered centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//Stephen_Kiprotich.jpg" width="100%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
<small>Image from Wikimedia Commons <a href="http://bit.ly/16kMKHQ" class="uri">http://bit.ly/16kMKHQ</a></small>
</td>
</tr>
</table>
<p>The first thing we will do is load a standard data set for regression modelling. The data consists of the pace of Olympic Gold Medal Marathon winners for the Olympics from 1896 to present. First we load in the data and plot.</p>
<div class="sourceCode" id="cb71"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb71-1"><a href="#cb71-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb71-2"><a href="#cb71-2" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> pods</span></code></pre></div>
<div class="sourceCode" id="cb72"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb72-1"><a href="#cb72-1" aria-hidden="true" tabindex="-1"></a>data <span class="op">=</span> pods.datasets.olympic_marathon_men()</span>
<span id="cb72-2"><a href="#cb72-2" aria-hidden="true" tabindex="-1"></a>x <span class="op">=</span> data[<span class="st">&#39;X&#39;</span>]</span>
<span id="cb72-3"><a href="#cb72-3" aria-hidden="true" tabindex="-1"></a>y <span class="op">=</span> data[<span class="st">&#39;Y&#39;</span>]</span>
<span id="cb72-4"><a href="#cb72-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb72-5"><a href="#cb72-5" aria-hidden="true" tabindex="-1"></a>offset <span class="op">=</span> y.mean()</span>
<span id="cb72-6"><a href="#cb72-6" aria-hidden="true" tabindex="-1"></a>scale <span class="op">=</span> np.sqrt(y.var())</span></code></pre></div>
<div class="figure">
<div id="olympic-marathon-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//datasets/olympic-marathon.svg" width="80%" style=" ">
</object>
</div>
<div id="olympic-marathon-magnify" class="magnify" onclick="magnifyFigure(&#39;olympic-marathon&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="olympic-marathon-caption" class="caption-frame">
<p>Figure: Olympic marathon pace times since 1896.</p>
</div>
</div>
<p>Things to notice about the data include the outlier in 1904, in this year, the olympics was in St Louis, USA. Organizational problems and challenges with dust kicked up by the cars following the race meant that participants got lost, and only very few participants completed.</p>
<p>More recent years see more consistently quick marathons.</p>
<h2 id="alan-turing">Alan Turing</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_ml/includes/alan-turing-marathon.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_ml/includes/alan-turing-marathon.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<div class="figure">
<div id="turing-run-times-figure" class="figure-frame">
<table>
<tr>
<td width="50%">
<div class="centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//turing-times.gif" width="100%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</td>
<td width="50%">
<div class="centered centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//turing-run.jpg" width="50%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</td>
</tr>
</table>
</div>
<div id="turing-run-times-magnify" class="magnify" onclick="magnifyFigure(&#39;turing-run-times&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="turing-run-times-caption" class="caption-frame">
<p>Figure: Alan Turing, in 1946 he was only 11 minutes slower than the winner of the 1948 games. Would he have won a hypothetical games held in 1946? Source: <a href="http://www.turing.org.uk/scrapbook/run.html" target="_blank">Alan Turing Internet Scrapbook</a>.</p>
</div>
</div>
<p>If we had to summarise the objectives of machine learning in one word, a very good candidate for that word would be <em>generalization</em>. What is generalization? From a human perspective it might be summarised as the ability to take lessons learned in one domain and apply them to another domain. If we accept the definition given in the first session for machine learning, <span class="math display">\[
\text{data} + \text{model} \stackrel{\text{compute}}{\rightarrow} \text{prediction}
\]</span> then we see that without a model we can’t generalise: we only have data. Data is fine for answering very specific questions, like “Who won the Olympic Marathon in 2012?” because we have that answer stored, however, we are not given the answer to many other questions. For example, Alan Turing was a formidable marathon runner, in 1946 he ran a time 2 hours 46 minutes (just under four minutes per kilometer, faster than I and most of the other <a href="http://www.parkrun.org.uk/sheffieldhallam/">Endcliffe Park Run</a> runners can do 5 km). What is the probability he would have won an Olympics if one had been held in 1946?</p>
<p>To answer this question we need to generalize, but before we formalize the concept of generalization let’s introduce some formal representation of what it means to generalize in machine learning.</p>
<h2 id="gaussian-process-fit">Gaussian Process Fit</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_gp/includes/olympic-marathon-gp.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_gp/includes/olympic-marathon-gp.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>Our first objective will be to perform a Gaussian process fit to the data, we’ll do this using the <a href="https://github.com/SheffieldML/GPy">GPy software</a>.</p>
<div class="sourceCode" id="cb73"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb73-1"><a href="#cb73-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> GPy</span></code></pre></div>
<div class="sourceCode" id="cb74"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb74-1"><a href="#cb74-1" aria-hidden="true" tabindex="-1"></a>m_full <span class="op">=</span> GPy.models.GPRegression(x,yhat)</span>
<span id="cb74-2"><a href="#cb74-2" aria-hidden="true" tabindex="-1"></a>_ <span class="op">=</span> m_full.optimize() <span class="co"># Optimize parameters of covariance function</span></span></code></pre></div>
<p>The first command sets up the model, then <code>m_full.optimize()</code> optimizes the parameters of the covariance function and the noise level of the model. Once the fit is complete, we’ll try creating some test points, and computing the output of the GP model in terms of the mean and standard deviation of the posterior functions between 1870 and 2030. We plot the mean function and the standard deviation at 200 locations. We can obtain the predictions using <code>y_mean, y_var = m_full.predict(xt)</code></p>
<div class="sourceCode" id="cb75"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb75-1"><a href="#cb75-1" aria-hidden="true" tabindex="-1"></a>xt <span class="op">=</span> np.linspace(<span class="dv">1870</span>,<span class="dv">2030</span>,<span class="dv">200</span>)[:,np.newaxis]</span>
<span id="cb75-2"><a href="#cb75-2" aria-hidden="true" tabindex="-1"></a>yt_mean, yt_var <span class="op">=</span> m_full.predict(xt)</span>
<span id="cb75-3"><a href="#cb75-3" aria-hidden="true" tabindex="-1"></a>yt_sd<span class="op">=</span>np.sqrt(yt_var)</span></code></pre></div>
<p>Now we plot the results using the helper function in <code>teaching_plots</code>.</p>
<div class="figure">
<div id="olympic-marathon-gp-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//gp/olympic-marathon-gp.svg" width="80%" style=" ">
</object>
</div>
<div id="olympic-marathon-gp-magnify" class="magnify" onclick="magnifyFigure(&#39;olympic-marathon-gp&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="olympic-marathon-gp-caption" class="caption-frame">
<p>Figure: Gaussian process fit to the Olympic Marathon data. The error bars are too large, perhaps due to the outlier from 1904.</p>
</div>
</div>
<h2 id="fit-quality">Fit Quality</h2>
<p>In the fit we see that the error bars (coming mainly from the noise variance) are quite large. This is likely due to the outlier point in 1904, ignoring that point we can see that a tighter fit is obtained. To see this make a version of the model, <code>m_clean</code>, where that point is removed.</p>
<div class="sourceCode" id="cb76"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb76-1"><a href="#cb76-1" aria-hidden="true" tabindex="-1"></a>x_clean<span class="op">=</span>np.vstack((x[<span class="dv">0</span>:<span class="dv">2</span>, :], x[<span class="dv">3</span>:, :]))</span>
<span id="cb76-2"><a href="#cb76-2" aria-hidden="true" tabindex="-1"></a>y_clean<span class="op">=</span>np.vstack((y[<span class="dv">0</span>:<span class="dv">2</span>, :], y[<span class="dv">3</span>:, :]))</span>
<span id="cb76-3"><a href="#cb76-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb76-4"><a href="#cb76-4" aria-hidden="true" tabindex="-1"></a>m_clean <span class="op">=</span> GPy.models.GPRegression(x_clean,y_clean)</span>
<span id="cb76-5"><a href="#cb76-5" aria-hidden="true" tabindex="-1"></a>_ <span class="op">=</span> m_clean.optimize()</span></code></pre></div>
<h2 id="deep-gp-fit">Deep GP Fit</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_deepgp/includes/olympic-marathon-deep-gp.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_deepgp/includes/olympic-marathon-deep-gp.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>Let’s see if a deep Gaussian process can help here. We will construct a deep Gaussian process with one hidden layer (i.e. one Gaussian process feeding into another).</p>
<p>Build a Deep GP with an additional hidden layer (one dimensional) to fit the model.</p>
<div class="sourceCode" id="cb77"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb77-1"><a href="#cb77-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> GPy</span>
<span id="cb77-2"><a href="#cb77-2" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> deepgp</span></code></pre></div>
<div class="sourceCode" id="cb78"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb78-1"><a href="#cb78-1" aria-hidden="true" tabindex="-1"></a>hidden <span class="op">=</span> <span class="dv">1</span></span>
<span id="cb78-2"><a href="#cb78-2" aria-hidden="true" tabindex="-1"></a>m <span class="op">=</span> deepgp.DeepGP([y.shape[<span class="dv">1</span>],hidden,x.shape[<span class="dv">1</span>]],Y<span class="op">=</span>yhat, X<span class="op">=</span>x, inits<span class="op">=</span>[<span class="st">&#39;PCA&#39;</span>,<span class="st">&#39;PCA&#39;</span>], </span>
<span id="cb78-3"><a href="#cb78-3" aria-hidden="true" tabindex="-1"></a>                  kernels<span class="op">=</span>[GPy.kern.RBF(hidden,ARD<span class="op">=</span><span class="va">True</span>),</span>
<span id="cb78-4"><a href="#cb78-4" aria-hidden="true" tabindex="-1"></a>                           GPy.kern.RBF(x.shape[<span class="dv">1</span>],ARD<span class="op">=</span><span class="va">True</span>)], <span class="co"># the kernels for each layer</span></span>
<span id="cb78-5"><a href="#cb78-5" aria-hidden="true" tabindex="-1"></a>                  num_inducing<span class="op">=</span><span class="dv">50</span>, back_constraint<span class="op">=</span><span class="va">False</span>)</span></code></pre></div>
<div class="sourceCode" id="cb79"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb79-1"><a href="#cb79-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> deepgp</span></code></pre></div>
<div class="sourceCode" id="cb80"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb80-1"><a href="#cb80-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Call the initalization</span></span>
<span id="cb80-2"><a href="#cb80-2" aria-hidden="true" tabindex="-1"></a>m.initialize()</span></code></pre></div>
<p>Now optimize the model.</p>
<div class="sourceCode" id="cb81"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb81-1"><a href="#cb81-1" aria-hidden="true" tabindex="-1"></a><span class="cf">for</span> layer <span class="kw">in</span> m.layers:</span>
<span id="cb81-2"><a href="#cb81-2" aria-hidden="true" tabindex="-1"></a>    layer.likelihood.variance.constrain_positive(warning<span class="op">=</span><span class="va">False</span>)</span>
<span id="cb81-3"><a href="#cb81-3" aria-hidden="true" tabindex="-1"></a>m.optimize(messages<span class="op">=</span><span class="va">True</span>,max_iters<span class="op">=</span><span class="dv">10000</span>)</span></code></pre></div>
<div class="sourceCode" id="cb82"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb82-1"><a href="#cb82-1" aria-hidden="true" tabindex="-1"></a>m.staged_optimize(messages<span class="op">=</span>(<span class="va">True</span>,<span class="va">True</span>,<span class="va">True</span>))</span></code></pre></div>
<h2 id="olympic-marathon-data-deep-gp">Olympic Marathon Data Deep GP</h2>
<div class="figure">
<div id="olympic-marathon-deep-gp-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//deepgp/olympic-marathon-deep-gp.svg" width="100%" style=" ">
</object>
</div>
<div id="olympic-marathon-deep-gp-magnify" class="magnify" onclick="magnifyFigure(&#39;olympic-marathon-deep-gp&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="olympic-marathon-deep-gp-caption" class="caption-frame">
<p>Figure: Deep GP fit to the Olympic marathon data. Error bars now change as the prediction evolves.</p>
</div>
</div>
<h2 id="olympic-marathon-data-deep-gp-1">Olympic Marathon Data Deep GP</h2>
<div class="figure">
<div id="olympic-marathon-deep-gp-samples-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//deepgp/olympic-marathon-deep-gp-samples.svg" width style=" ">
</object>
</div>
<div id="olympic-marathon-deep-gp-samples-magnify" class="magnify" onclick="magnifyFigure(&#39;olympic-marathon-deep-gp-samples&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="olympic-marathon-deep-gp-samples-caption" class="caption-frame">
<p>Figure: Point samples run through the deep Gaussian process show the distribution of output locations.</p>
</div>
</div>
<h2 id="fitted-gp-for-each-layer">Fitted GP for each layer</h2>
<p>Now we explore the GPs the model has used to fit each layer. First of all, we look at the hidden layer.</p>
<div class="figure">
<div id="olympic-marathon-deep-gp-layer-0-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//deepgp/olympic-marathon-deep-gp-layer-0.svg" width style=" ">
</object>
</div>
<div id="olympic-marathon-deep-gp-layer-0-magnify" class="magnify" onclick="magnifyFigure(&#39;olympic-marathon-deep-gp-layer-0&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="olympic-marathon-deep-gp-layer-0-caption" class="caption-frame">
<p>Figure: The mapping from input to the latent layer is broadly, with some flattening as time goes on. Variance is high across the input range.</p>
</div>
</div>
<div class="figure">
<div id="olympic-marathon-deep-gp-layer-1-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//deepgp/olympic-marathon-deep-gp-layer-1.svg" width style=" ">
</object>
</div>
<div id="olympic-marathon-deep-gp-layer-1-magnify" class="magnify" onclick="magnifyFigure(&#39;olympic-marathon-deep-gp-layer-1&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="olympic-marathon-deep-gp-layer-1-caption" class="caption-frame">
<p>Figure: The mapping from the latent layer to the output layer.</p>
</div>
</div>
<h2 id="olympic-marathon-pinball-plot">Olympic Marathon Pinball Plot</h2>
<div class="figure">
<div id="olympic-marathon-deep-gp-pinball-figure" class="figure-frame">
<object class="svgplot " data="https://inverseprobability.com/slides/diagrams//deepgp/olympic-marathon-deep-gp-pinball.svg" width="80%" style=" ">
</object>
</div>
<div id="olympic-marathon-deep-gp-pinball-magnify" class="magnify" onclick="magnifyFigure(&#39;olympic-marathon-deep-gp-pinball&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="olympic-marathon-deep-gp-pinball-caption" class="caption-frame">
<p>Figure: A pinball plot shows the movement of the ‘ball’ as it passes through each layer of the Gaussian processes. Mean directions of movement are shown by lines. Shading gives one standard deviation of movement position. At each layer, the uncertainty is reset. The overal uncertainty is the cumulative uncertainty from all the layers. There is some grouping of later points towards the right in the first layer, which also injects a large amount of uncertainty. Due to flattening of the curve in the second layer towards the right the uncertainty is reduced in final output.</p>
</div>
</div>
<p>The pinball plot shows the flow of any input ball through the deep Gaussian process. In a pinball plot a series of vertical parallel lines would indicate a purely linear function. For the olypmic marathon data we can see the first layer begins to shift from input towards the right. Note it also does so with some uncertainty (indicated by the shaded backgrounds). The second layer has less uncertainty, but bunches the inputs more strongly to the right. This input layer of uncertainty, followed by a layer that pushes inputs to the right is what gives the heteroschedastic noise.</p>
<h2 id="mxfusion-modular-probabilistic-programming-on-mxnet">MXFusion: Modular Probabilistic Programming on MXNet</h2>
<div style="text-align:right">
<span class="editsection-bracket" style="">[</span><span class="editsection" style=""><a href="https://github.com/lawrennd/snippets/edit/main/_ml/includes/mxfusion-intro.md" target="_blank" onclick="ga('send', 'event', 'Edit Page', 'Edit', 'https://github.com/lawrennd/snippets/edit/main/_ml/includes/mxfusion-intro.md', 13);">edit</a></span><span class="editsection-bracket" style="">]</span>
</div>
<p>One challenge for practitioners in Gaussian processes, is flexible software that allows the construction of the relevant GP modle. With this in mind, the Amazon Cambridge team has developed MXFusion. It is a modular probabilistic programming language focussed on efficient implementation of hybrid GP-neural network models, but with additional probabilistic programming capabilities.</p>
<div class="figure">
<div id="mxfusion-software-figure" class="figure-frame">
<div class="centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//ml/mxfusion.png" width="70%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</div>
<div id="mxfusion-software-magnify" class="magnify" onclick="magnifyFigure(&#39;mxfusion-software&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="mxfusion-software-caption" class="caption-frame">
<p>Figure: MXFusion is a probabilistic programming language targeted specifically at Gaussian process models and combining them with probaiblistic neural network. It is available through the MIT license and we welcome contributions throguh the Github repository <a href="https://github.com/amzn/MXFusion" class="uri">https://github.com/amzn/MXFusion</a>.</p>
</div>
</div>
<p>We developed the framework for greater ease of transitioning models from ‘science’ to ‘production,’ our aim was to have code that could be created by scientists, but deployed in our systems through solutions such as AWS SageMaker.</p>
<p>\ericMeissner{15%}\zhenwenDai{15%}</p>
<div class="figure">
<div id="mxfusion-software-logo-figure" class="figure-frame">
<table>
<tr>
<td width="70%">
<ul>
<li>Work by Eric Meissner and Zhenwen Dai.</li>
<li>Probabilistic programming.</li>
<li>Available on <a href="https://github.com/amzn/mxfusion">Github</a></li>
</ul>
</td>
<td width="30%">
<div class="centered" style="">
<img class="" src="https://inverseprobability.com/slides/diagrams//mxfusion-logo.png" width="80%" height="auto" align="center" style="background:none; border:none; box-shadow:none; display:block; margin-left:auto; margin-right:auto;vertical-align:middle">
</div>
</td>
</tr>
</table>
</div>
<div id="mxfusion-software-logo-magnify" class="magnify" onclick="magnifyFigure(&#39;mxfusion-software-logo&#39;)">
<img class="img-button" src="{{ '/assets/images/Magnify_Large.svg' | relative_url }}" style="width:1.5ex">
</div>
<div id="mxfusion-software-logo-caption" class="caption-frame">
<p>Figure: The MXFusion software.</p>
</div>
</div>
<h2 id="conclusion">Conclusion</h2>
<p>Machine learning models are deployed as components in an interacting system to achieve modern AI. Some of those components are inspired by a mechanistic understanding of the world around us (e.g. economic or physical understanding).</p>
<p>Meta modelling involves fiting machine learning models to existing systems to improve speed and interpretability.</p>
<p>Deep Gaussian processes are a flexible approach to meta modelling, which provide the necessary uncertainty estimates and the potential for being more mathematically tractable.</p>
<h2 id="thanks">Thanks!</h2>
<p>For more information on these subjects and more you might want to check the following resources.</p>
<ul>
<li>twitter: <a href="https://twitter.com/lawrennd">@lawrennd</a></li>
<li>podcast: <a href="http://thetalkingmachines.com">The Talking Machines</a></li>
<li>newspaper: <a href="http://www.theguardian.com/profile/neil-lawrence">Guardian Profile Page</a></li>
<li>blog: <a href="http://inverseprobability.com/blog.html">http://inverseprobability.com</a></li>
</ul>
<h1 class="unnumbered" id="references">References</h1>
<div id="refs" class="references csl-bib-body hanging-indent" role="doc-bibliography">
<div id="ref-Dunlop:deep2017" class="csl-entry" role="doc-biblioentry">
Dunlop, M.M., Girolami, M.A., Stuart, A.M., Teckentrup, A.L., n.d. How deep are deep <span>G</span>aussian processes? Journal of Machine Learning Research 19, 1–46.
</div>
<div id="ref-Perdikaris:multifidelity17" class="csl-entry" role="doc-biblioentry">
Perdikaris, P., Raissi, M., Damianou, A., Lawrence, N.D., Karniadakis, G.E., 2017. Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling. Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 473. <a href="https://doi.org/10.1098/rspa.2016.0751">https://doi.org/10.1098/rspa.2016.0751</a>
</div>
<div id="ref-Shahriari-human16" class="csl-entry" role="doc-biblioentry">
Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., de Freitas, N., 2016. Taking the human out of the loop: A review of <span>B</span>ayesian optimization. Proceedings of the IEEE 104, 148–175. <a href="https://doi.org/10.1109/JPROC.2015.2494218">https://doi.org/10.1109/JPROC.2015.2494218</a>
</div>
<div id="ref-Taigman:deepface14" class="csl-entry" role="doc-biblioentry">
Taigman, Y., Yang, M., Ranzato, M., Wolf, L., 2014. <span>DeepFace</span>: Closing the gap to human-level performance in face verification, in: Proceedings of the <span>IEEE</span> Computer Society Conference on Computer Vision and Pattern Recognition. <a href="https://doi.org/10.1109/CVPR.2014.220">https://doi.org/10.1109/CVPR.2014.220</a>
</div>
</div>
<section class="footnotes" role="doc-endnotes">
<hr />
<ol>
<li id="fn1" role="doc-endnote"><p>Apple has fixed this issue so that Siri no longer does this.<a href="#fnref1" class="footnote-back" role="doc-backlink">↩︎</a></p></li>
</ol>
</section>