-
Notifications
You must be signed in to change notification settings - Fork 16
/
Copy pathresource-groups.html
280 lines (267 loc) · 19.6 KB
/
resource-groups.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>4.5. Resource Groups — Presto 0.202 Documentation</title>
<link rel="stylesheet" href="../_static/presto.css" type="text/css" />
<link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '0.202',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="../_static/jquery.js"></script>
<script type="text/javascript" src="../_static/underscore.js"></script>
<script type="text/javascript" src="../_static/doctools.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="top" title="Presto 0.202 Documentation" href="../index.html" />
<link rel="up" title="4. Administration" href="../admin.html" />
<link rel="next" title="5. Connectors" href="../connector.html" />
<link rel="prev" title="4.4. Spill to Disk" href="spill.html" />
</head>
<body role="document">
<div class="header">
<h1 class="heading"><a href="../index.html">
<span>Presto 0.202 Documentation</span></a></h1>
<h2 class="heading"><span>4.5. Resource Groups</span></h2>
</div>
<div class="topnav">
<p class="nav">
<span class="left">
« <a href="spill.html">4.4. Spill to Disk</a>
</span>
<span class="right">
<a href="../connector.html">5. Connectors</a> »
</span>
</p>
</div>
<div class="content">
<div class="section" id="resource-groups">
<h1>4.5. Resource Groups</h1>
<p>Resource groups place limits on resource usage, and can enforce queueing policies on
queries that run within them or divide their resources among sub groups. A query
belongs to a single resource group, and consumes resources from that group (and its ancestors).
Except for the limit on queued queries, when a resource group runs out of a resource
it does not cause running queries to fail; instead new queries become queued.
A resource group may have sub groups or may accept queries, but may not do both.</p>
<p>The resource groups and associated selection rules are configured by a manager which is pluggable.
Add an <code class="docutils literal"><span class="pre">etc/resource-groups.properties</span></code> file with the following contents to enable
the built-in manager that reads a JSON config file:</p>
<div class="highlight-none"><div class="highlight"><pre><span></span>resource-groups.configuration-manager=file
resource-groups.config-file=etc/resource_groups.json
</pre></div>
</div>
<p>Change the value of <code class="docutils literal"><span class="pre">resource-groups.config-file</span></code> to point to a JSON config file,
which can be an absolute path, or a path relative to the Presto data directory.</p>
<div class="section" id="resource-group-properties">
<h2>Resource Group Properties</h2>
<ul>
<li><p class="first"><code class="docutils literal"><span class="pre">name</span></code> (required): name of the group. May be a template (see below).</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">maxQueued</span></code> (required): maximum number of queued queries. Once this limit is reached
new queries will be rejected.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">hardConcurrencyLimit</span></code> (required): maximum number of running queries.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">softMemoryLimit</span></code> (required): maximum amount of distributed memory this
group may use before new queries become queued. May be specified as
an absolute value (i.e. <code class="docutils literal"><span class="pre">1GB</span></code>) or as a percentage (i.e. <code class="docutils literal"><span class="pre">10%</span></code>) of the cluster’s memory.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">softCpuLimit</span></code> (optional): maximum amount of CPU time this
group may use in a period (see <code class="docutils literal"><span class="pre">cpuQuotaPeriod</span></code>) before a penalty will be applied to
the maximum number of running queries. <code class="docutils literal"><span class="pre">hardCpuLimit</span></code> must also be specified.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">hardCpuLimit</span></code> (optional): maximum amount of CPU time this
group may use in a period.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">schedulingPolicy</span></code> (optional): specifies how queued queries are selected to run,
and how sub groups become eligible to start their queries. May be one of three values:</p>
<blockquote>
<div><ul class="simple">
<li><code class="docutils literal"><span class="pre">fair</span></code> (default): queued queries are processed first-in-first-out, and sub groups
must take turns starting new queries (if they have any queued).</li>
<li><code class="docutils literal"><span class="pre">weighted</span></code>: queued queries are selected stochastically in proportion to their priority
(specified via the <code class="docutils literal"><span class="pre">query_priority</span></code> <a class="reference internal" href="../sql/set-session.html"><span class="doc">session property</span></a>). Sub groups are selected
to start new queries in proportion to their <code class="docutils literal"><span class="pre">schedulingWeight</span></code>.</li>
<li><code class="docutils literal"><span class="pre">query_priority</span></code>: all sub groups must also be configured with <code class="docutils literal"><span class="pre">query_priority</span></code>.
Queued queries will be selected strictly according to their priority.</li>
</ul>
</div></blockquote>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">schedulingWeight</span></code> (optional): weight of this sub group. See above.
Defaults to <code class="docutils literal"><span class="pre">1</span></code>.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">jmxExport</span></code> (optional): If true, group statistics are exported to JMX for monitoring.
Defaults to <code class="docutils literal"><span class="pre">false</span></code>.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">queuedTimeLimit</span></code> (optional): maximum amount of time a query may be in the queue for this group
before it is marked as failed.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">runningTimeLimit</span></code> (optional): maximum amount of time a query in this group can execute
(not including queued time) before it is marked as failed.</p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">subGroups</span></code> (optional): list of sub groups.</p>
</li>
</ul>
</div>
<div class="section" id="selector-properties">
<h2>Selector Properties</h2>
<ul>
<li><p class="first"><code class="docutils literal"><span class="pre">user</span></code> (optional): regex to match against user name. Defaults to <code class="docutils literal"><span class="pre">.*</span></code></p>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">source</span></code> (optional): regex to match against source string. Defaults to <code class="docutils literal"><span class="pre">.*</span></code></p>
</li>
<li><dl class="first docutils">
<dt><code class="docutils literal"><span class="pre">queryType</span></code> (optional): string to match against the type of the query submitted. The query type can be:</dt>
<dd><ul class="first last simple">
<li><code class="docutils literal"><span class="pre">DATA_DEFINITION</span></code>: Queries that alter/create/drop the metadata of schemas/tables/views, and that manage
prepared statements, privileges, sessions, and transactions.</li>
<li><code class="docutils literal"><span class="pre">DELETE</span></code>: <code class="docutils literal"><span class="pre">DELETE</span></code> queries.</li>
<li><code class="docutils literal"><span class="pre">DESCRIBE</span></code>: <code class="docutils literal"><span class="pre">DESCRIBE</span></code>, <code class="docutils literal"><span class="pre">DESCRIBE</span> <span class="pre">INPUT</span></code>, <code class="docutils literal"><span class="pre">DESCRIBE</span> <span class="pre">OUTPUT</span></code>, and <code class="docutils literal"><span class="pre">SHOW</span></code> queries.</li>
<li><code class="docutils literal"><span class="pre">EXPLAIN</span></code>: <code class="docutils literal"><span class="pre">EXPLAIN</span></code> queries.</li>
<li><code class="docutils literal"><span class="pre">INSERT</span></code>: <code class="docutils literal"><span class="pre">INSERT</span></code> and <code class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span> <span class="pre">AS</span> <span class="pre">SELECT</span></code> queries.</li>
<li><code class="docutils literal"><span class="pre">SELECT</span></code>: <code class="docutils literal"><span class="pre">SELECT</span></code> queries.</li>
</ul>
</dd>
</dl>
</li>
<li><p class="first"><code class="docutils literal"><span class="pre">group</span></code> (required): the group these queries will run in.</p>
</li>
</ul>
</div>
<div class="section" id="global-properties">
<h2>Global Properties</h2>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">cpuQuotaPeriod</span></code> (optional): the period in which cpu quotas are enforced.</li>
</ul>
<p>Selectors are processed sequentially and the first one that matches will be used.
In the example configuration below, there are five resource group templates.
In the <code class="docutils literal"><span class="pre">adhoc_${USER}</span></code> group, <code class="docutils literal"><span class="pre">${USER}</span></code> will be expanded to the name of the
user that submitted the query. <code class="docutils literal"><span class="pre">${SOURCE}</span></code> is also supported, which expands
to the source submitting the query. The source name can be set as follows:</p>
<blockquote>
<div><ul class="simple">
<li>CLI: use the <code class="docutils literal"><span class="pre">--source</span></code> option.</li>
<li>JDBC: set the <code class="docutils literal"><span class="pre">ApplicationName</span></code> client info property on the <code class="docutils literal"><span class="pre">Connection</span></code> instance.</li>
</ul>
</div></blockquote>
<p>There are three selectors that define which queries run in which resource group:</p>
<blockquote>
<div><ul class="simple">
<li>The first selector places queries from <code class="docutils literal"><span class="pre">bob</span></code> into the admin group.</li>
<li>The second selector states that all data definition queries that come from a source that includes “pipeline”
should run in the user’s personal data definition group, which belongs to the
<code class="docutils literal"><span class="pre">globa.data_definition</span></code> parent group.</li>
<li>The third selector states that all queries that come from a source that includes “pipeline”
should run in the user’s personal pipeline group, which belongs to the <code class="docutils literal"><span class="pre">global.pipeline</span></code>
parent group.</li>
<li>The last selector is a catch all, which puts all queries into the user’s adhoc group.</li>
</ul>
</div></blockquote>
<p>All together these selectors implement the policy that <code class="docutils literal"><span class="pre">bob</span></code> is an admin and
all other users are subject to the following limits:</p>
<blockquote>
<div><ul class="simple">
<li>Users are allowed to have up to 2 adhoc queries running. Additionally, they may run one pipeline.</li>
<li>No more than 5 “pipeline” queries may run at once.</li>
<li>No more than 100 total queries may run at once, unless they’re from the admin.</li>
</ul>
</div></blockquote>
<div class="highlight-json"><div class="highlight"><pre><span></span><span class="p">{</span>
<span class="nt">"rootGroups"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"global"</span><span class="p">,</span>
<span class="nt">"softMemoryLimit"</span><span class="p">:</span> <span class="s2">"80%"</span><span class="p">,</span>
<span class="nt">"hardConcurrencyLimit"</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span>
<span class="nt">"maxQueued"</span><span class="p">:</span> <span class="mi">1000</span><span class="p">,</span>
<span class="nt">"schedulingPolicy"</span><span class="p">:</span> <span class="s2">"weighted"</span><span class="p">,</span>
<span class="nt">"jmxExport"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="nt">"subGroups"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"data_definition_${USER}"</span><span class="p">,</span>
<span class="nt">"softMemoryLimit"</span><span class="p">:</span> <span class="s2">"10%"</span><span class="p">,</span>
<span class="nt">"hardConcurrencyLimit"</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
<span class="nt">"maxQueued"</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span>
<span class="nt">"schedulingWeight"</span><span class="p">:</span> <span class="mi">1</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"adhoc_${USER}"</span><span class="p">,</span>
<span class="nt">"softMemoryLimit"</span><span class="p">:</span> <span class="s2">"10%"</span><span class="p">,</span>
<span class="nt">"hardConcurrencyLimit"</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
<span class="nt">"maxQueued"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="nt">"schedulingWeight"</span><span class="p">:</span> <span class="mi">9</span><span class="p">,</span>
<span class="nt">"schedulingPolicy"</span><span class="p">:</span> <span class="s2">"query_priority"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"pipeline"</span><span class="p">,</span>
<span class="nt">"softMemoryLimit"</span><span class="p">:</span> <span class="s2">"20%"</span><span class="p">,</span>
<span class="nt">"hardConcurrencyLimit"</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span>
<span class="nt">"maxQueued"</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span>
<span class="nt">"schedulingWeight"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="nt">"jmxExport"</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="nt">"subGroups"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"pipeline_${USER}"</span><span class="p">,</span>
<span class="nt">"softMemoryLimit"</span><span class="p">:</span> <span class="s2">"10%"</span><span class="p">,</span>
<span class="nt">"hardConcurrencyLimit"</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="nt">"maxQueued"</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span>
<span class="nt">"schedulingPolicy"</span><span class="p">:</span> <span class="s2">"query_priority"</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"admin"</span><span class="p">,</span>
<span class="nt">"softMemoryLimit"</span><span class="p">:</span> <span class="s2">"100%"</span><span class="p">,</span>
<span class="nt">"hardConcurrencyLimit"</span><span class="p">:</span> <span class="mi">200</span><span class="p">,</span>
<span class="nt">"maxQueued"</span><span class="p">:</span> <span class="mi">100</span><span class="p">,</span>
<span class="nt">"schedulingPolicy"</span><span class="p">:</span> <span class="s2">"query_priority"</span><span class="p">,</span>
<span class="nt">"jmxExport"</span><span class="p">:</span> <span class="kc">true</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="nt">"selectors"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"user"</span><span class="p">:</span> <span class="s2">"bob"</span><span class="p">,</span>
<span class="nt">"group"</span><span class="p">:</span> <span class="s2">"admin"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"source"</span><span class="p">:</span> <span class="s2">".*pipeline.*"</span><span class="p">,</span>
<span class="nt">"queryType"</span><span class="p">:</span> <span class="s2">"DATA_DEFINITION"</span><span class="p">,</span>
<span class="nt">"group"</span><span class="p">:</span> <span class="s2">"global.data_definition_${USER}"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"source"</span><span class="p">:</span> <span class="s2">".*pipeline.*"</span><span class="p">,</span>
<span class="nt">"group"</span><span class="p">:</span> <span class="s2">"global.pipeline.pipeline_${USER}"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"group"</span><span class="p">:</span> <span class="s2">"global.adhoc_${USER}"</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="nt">"cpuQuotaPeriod"</span><span class="p">:</span> <span class="s2">"1h"</span>
<span class="p">}</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="bottomnav">
<p class="nav">
<span class="left">
« <a href="spill.html">4.4. Spill to Disk</a>
</span>
<span class="right">
<a href="../connector.html">5. Connectors</a> »
</span>
</p>
</div>
<div class="footer" role="contentinfo">
</div>
</body>
</html>