0.191/overview/concepts.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">


<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>1.2. Presto Concepts &#8212; Presto 0.191 Documentation</title>
    
    <link rel="stylesheet" href="../_static/presto.css" type="text/css" />
    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../',
        VERSION:     '0.191',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="../_static/jquery.js"></script>
    <script type="text/javascript" src="../_static/underscore.js"></script>
    <script type="text/javascript" src="../_static/doctools.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="top" title="Presto 0.191 Documentation" href="../index.html" />
    <link rel="up" title="1. Overview" href="../overview.html" />
    <link rel="next" title="2. Installation" href="../installation.html" />
    <link rel="prev" title="1.1. Use Cases" href="use-cases.html" /> 
  </head>
  <body role="document">
<div class="header">
    <h1 class="heading"><a href="../index.html">
        <span>Presto 0.191 Documentation</span></a></h1>
    <h2 class="heading"><span>1.2. Presto Concepts</span></h2>
</div>
<div class="topnav">
    
<p class="nav">
    <span class="left">
        &laquo; <a href="use-cases.html">1.1. Use Cases</a>
    </span>
    <span class="right">
        <a href="../installation.html">2. Installation</a> &raquo;
    </span>
</p>

</div>
<div class="content">
    
  <div class="section" id="presto-concepts">
<h1>1.2. Presto Concepts</h1>
<div class="contents local topic" id="contents">
<ul class="simple">
<li><a class="reference internal" href="#overview" id="id1">Overview</a></li>
<li><a class="reference internal" href="#server-types" id="id2">Server Types</a></li>
<li><a class="reference internal" href="#data-sources" id="id3">Data Sources</a></li>
<li><a class="reference internal" href="#query-execution-model" id="id4">Query Execution Model</a></li>
</ul>
</div>
<div class="section" id="overview">
<h2>Overview</h2>
<p>To understand Presto you must first understand the terms and concepts
used throughout the Presto documentation.</p>
<p>While it&#8217;s easy to understand statements and queries, as an end-user
you should have familiarity with concepts such as stages and splits to
take full advantage of Presto to execute efficient queries.  As a
Presto administrator or a Presto contributor you should understand how
Presto&#8217;s concepts of stages map to tasks and how tasks contain a set
of drivers which process data.</p>
<p>This section provides a solid definition for the core concepts
referenced throughout Presto, and these sections are sorted from most
general to most specific.</p>
</div>
<div class="section" id="server-types">
<h2>Server Types</h2>
<p>There are two types of Presto servers: coordinators and workers. The
following section explains the difference between the two.</p>
<div class="section" id="coordinator">
<h3>Coordinator</h3>
<p>The Presto coordinator is the server that is responsible for parsing
statements, planning queries, and managing Presto worker nodes.  It is
the &#8220;brain&#8221; of a Presto installation and is also the node to which a
client connects to submit statements for execution. Every Presto
installation must have a Presto coordinator alongside one or more
Presto workers. For development or testing purposes, a single
instance of Presto can be configured to perform both roles.</p>
<p>The coordinator keeps track of the activity on each worker and
coordinates the execution of a query. The coordinator creates
a logical model of a query involving a series of stages which is then
translated into a series of connected tasks running on a cluster of
Presto workers.</p>
<p>Coordinators communicate with workers and clients using a REST API.</p>
</div>
<div class="section" id="worker">
<h3>Worker</h3>
<p>A Presto worker is a server in a Presto installation which is responsible
for executing tasks and processing data. Worker nodes fetch data from
connectors and exchange intermediate data with each other. The coordinator
is responsible for fetching results from the workers and returning the
final results to the client.</p>
<p>When a Presto worker process starts up, it advertises itself to the discovery
server in the coordinator, which makes it available to the Presto coordinator
for task execution.</p>
<p>Workers communicate with other workers and Presto coordinators
using a REST API.</p>
</div>
</div>
<div class="section" id="data-sources">
<h2>Data Sources</h2>
<p>Throughout this documentation, you&#8217;ll read terms such as connector,
catalog, schema, and table. These fundamental concepts cover Presto&#8217;s
model of a particular data source and are described in the following
section.</p>
<div class="section" id="connector">
<h3>Connector</h3>
<p>A connector adapts Presto to a data source such as Hive or a
relational database. You can think of a connector the same way you
think of a driver for a database. It is an implementation of Presto&#8217;s
<a class="reference internal" href="../develop/spi-overview.html"><span class="doc">SPI</span></a> which allows Presto to interact
with a resource using a standard API.</p>
<p>Presto contains several built-in connectors: a connector for
<a class="reference internal" href="../connector/jmx.html"><span class="doc">JMX</span></a>, a <a class="reference internal" href="../connector/system.html"><span class="doc">System</span></a>
connector which provides access to built-in system tables,
a <a class="reference internal" href="../connector/hive.html"><span class="doc">Hive</span></a> connector, and a
<a class="reference internal" href="../connector/tpch.html"><span class="doc">TPCH</span></a> connector designed to serve TPC-H benchmark
data. Many third-party developers have contributed connectors so that
Presto can access data in a variety of data sources.</p>
<p>Every catalog is associated with a specific connector. If you examine
a catalog configuration file, you will see that each contains a
mandatory property <code class="docutils literal"><span class="pre">connector.name</span></code> which is used by the catalog
manager to create a connector for a given catalog. It is possible
to have more than one catalog use the same connector to access two
different instances of a similar database. For example, if you have
two Hive clusters, you can configure two catalogs in a single Presto
cluster that both use the Hive connector, allowing you to query data
from both Hive clusters (even within the same SQL query).</p>
</div>
<div class="section" id="catalog">
<h3>Catalog</h3>
<p>A Presto catalog contains schemas and references a data source via a
connector.  For example, you can configure a JMX catalog to provide
access to JMX information via the JMX connector. When you run a SQL
statement in Presto, you are running it against one or more catalogs.
Other examples of catalogs include the Hive catalog to connect to a
Hive data source.</p>
<p>When addressing a table in Presto, the fully-qualified table name is
always rooted in a catalog. For example, a fully-qualified table name
of <code class="docutils literal"><span class="pre">hive.test_data.test</span></code> would refer to the <code class="docutils literal"><span class="pre">test</span></code> table in the
<code class="docutils literal"><span class="pre">test_data</span></code> schema in the <code class="docutils literal"><span class="pre">hive</span></code> catalog.</p>
<p>Catalogs are defined in properties files stored in the Presto
configuration directory.</p>
</div>
<div class="section" id="schema">
<h3>Schema</h3>
<p>Schemas are a way to organize tables. Together, a catalog and schema
define a set of tables that can be queried. When accessing Hive or a
relational database such as MySQL with Presto, a schema translates to
the same concept in the target database. Other types of connectors may
choose to organize tables into schemas in a way that makes sense for
the underlying data source.</p>
</div>
<div class="section" id="table">
<h3>Table</h3>
<p>A table is a set of unordered rows which are organized into named columns
with types. This is the same as in any relational database. The mapping
from source data to tables is defined by the connector.</p>
</div>
</div>
<div class="section" id="query-execution-model">
<h2>Query Execution Model</h2>
<p>Presto executes SQL statements and turns these statements into queries
that are executed across a distributed cluster of coordinator and workers.</p>
<div class="section" id="statement">
<h3>Statement</h3>
<p>Presto executes ANSI-compatible SQL statements.  When the Presto
documentation refers to a statement, it is referring to statements as
defined in the ANSI SQL standard which consists of clauses,
expressions, and predicates.</p>
<p>Some readers might be curious why this section lists seperate concepts
for statements and queries. This is necessary because, in Presto,
statements simply refer to the textual representation of a SQL
statement. When a statement is executed, Presto creates a query along
with a query plan that is then distributed across a series of Presto
workers.</p>
</div>
<div class="section" id="query">
<h3>Query</h3>
<p>When Presto parses a statement, it converts it into a query and creates
a distributed query plan which is then realized as a series of
interconnected stages running on Presto workers. When you retrieve
information about a query in Presto, you receive a snapshot of every
component that is involved in producing a result set in response to a
statement.</p>
<p>The difference between a statement and a query is simple. A statement
can be thought of as the SQL text that is passed to Presto, while a query
refers to the configuration and components instantiated to execute
that statement. A query encompasses stages, tasks, splits, connectors,
and other components and data sources working in concert to produce a
result.</p>
</div>
<div class="section" id="stage">
<h3>Stage</h3>
<p>When Presto executes a query, it does so by breaking up the execution
into a hierarchy of stages. For example, if Presto needs to aggregate
data from one billion rows stored in Hive, it does so by creating a
root stage to aggregate the output of several other stages all of
which are designed to implement different sections of a distributed
query plan.</p>
<p>The hierarchy of stages that comprises a query resembles a tree.
Every query has a root stage which is responsible for aggregating
the output from other stages. Stages are what the coordinator uses to
model a distributed query plan, but stages themselves don&#8217;t run on
Presto workers.</p>
</div>
<div class="section" id="task">
<h3>Task</h3>
<p>As mentioned in the previous section, stages model a particular
section of a distributed query plan, but stages themselves don&#8217;t
execute on Presto workers. To understand how a stage is executed,
you&#8217;ll need to understand that a stage is implemented as a series of
tasks distributed over a network of Presto workers.</p>
<p>Tasks are the &#8220;work horse&#8221; in the Presto architecture as a distributed
query plan is deconstructed into a series of stages which are then
translated to tasks which then act upon or process splits. A Presto
task has inputs and outputs, and just as a stage can be executed in
parallel by a series of tasks, a task is executing in parallel with a
series of drivers.</p>
</div>
<div class="section" id="split">
<h3>Split</h3>
<p>Tasks operate on splits which are sections of a larger data
set. Stages at the lowest level of a distributed query plan retrieve
data via splits from connectors, and intermediate stages at a higher
level of a distributed query plan retrieve data from other stages.</p>
<p>When Presto is scheduling a query, the coordinator will query a
connector for a list of all splits that are available for a table.
The coordinator keeps track of which machines are running which tasks
and what splits are being processed by which tasks.</p>
</div>
<div class="section" id="driver">
<h3>Driver</h3>
<p>Tasks contain one or more parallel drivers. Drivers act upon data and
combine operators to produce output that is then aggregated by a task
and then delivered to another task in a another stage. A driver is a
sequence of operator instances, or you can think of a driver as a
physical set of operators in memory. It is the lowest level of
parallelism in the Presto architecture. A driver has one input and
one output.</p>
</div>
<div class="section" id="operator">
<h3>Operator</h3>
<p>An operator consumes, transforms and produces data. For example, a table
scan fetches data from a connector and produces data that can be consumed
by other operators, and a filter operator consumes data and produces a
subset by applying a predicate over the input data.</p>
</div>
<div class="section" id="exchange">
<h3>Exchange</h3>
<p>Exchanges transfer data between Presto nodes for different stages of
a query. Tasks produce data into an output buffer and consume data
from other tasks using an exchange client.</p>
</div>
</div>
</div>


</div>
<div class="bottomnav">
    
<p class="nav">
    <span class="left">
        &laquo; <a href="use-cases.html">1.1. Use Cases</a>
    </span>
    <span class="right">
        <a href="../installation.html">2. Installation</a> &raquo;
    </span>
</p>

</div>

    <div class="footer" role="contentinfo">
    </div>
  </body>
</html>