forked from mpdehaan/vespene-io.github.io
/
development_guide.html
374 lines (257 loc) · 16.9 KB
/
development_guide.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Development Guide — Vespene documentation</title>
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="FAQ/Troubleshooting" href="faq.html" />
<link rel="prev" title="Development Setup" href="development_setup.html" />
<script src="_static/js/modernizr.min.js"></script>
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search">
<a href="contents.html" class="icon icon-home"> Vespene
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<p class="caption"><span class="caption-text">Getting Started</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="index.html">About Vespene</a></li>
<li class="toctree-l1"><a class="reference internal" href="setup.html">Setup Guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
</ul>
<p class="caption"><span class="caption-text">Fundmentals</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="workers.html">Workers</a></li>
<li class="toctree-l1"><a class="reference internal" href="projects.html">Projects</a></li>
<li class="toctree-l1"><a class="reference internal" href="variables.html">Variables</a></li>
<li class="toctree-l1"><a class="reference internal" href="access.html">Access</a></li>
</ul>
<p class="caption"><span class="caption-text">Workflow</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="importing.html">Imports (.vespene)</a></li>
<li class="toctree-l1"><a class="reference internal" href="launch_questions.html">Launch Questions</a></li>
<li class="toctree-l1"><a class="reference internal" href="pipelines.html">Pipelines</a></li>
<li class="toctree-l1"><a class="reference internal" href="scheduling.html">Scheduling</a></li>
<li class="toctree-l1"><a class="reference internal" href="webhooks.html">Webhooks</a></li>
</ul>
<p class="caption"><span class="caption-text">Admin</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="authz.html">Authorization</a></li>
<li class="toctree-l1"><a class="reference internal" href="cli.html">CLI</a></li>
<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="settings.html">Settings</a></li>
<li class="toctree-l1"><a class="reference internal" href="upgrades.html">Upgrades</a></li>
</ul>
<p class="caption"><span class="caption-text">Community</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="resources.html">Resources</a></li>
<li class="toctree-l1"><a class="reference internal" href="development_setup.html">Development Setup</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Development Guide</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#architecture">Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="#code-structure">Code Structure</a></li>
<li class="toctree-l2"><a class="reference internal" href="#code-conventions">Code Conventions</a></li>
<li class="toctree-l2"><a class="reference internal" href="#javascript-static-file-changes">Javascript & Static File Changes</a></li>
<li class="toctree-l2"><a class="reference internal" href="#database-changes">Database Changes</a></li>
<li class="toctree-l2"><a class="reference internal" href="#adding-plugins">Adding Plugins</a></li>
<li class="toctree-l2"><a class="reference internal" href="#dynamic-imports">Dynamic Imports</a></li>
<li class="toctree-l2"><a class="reference internal" href="#debugging-tips">Debugging Tips</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ / Troubleshooting</a></li>
<li class="toctree-l1"><a class="reference internal" href="partnership.html">Partnership Program</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
<nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="contents.html">Vespene</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
<ul class="wy-breadcrumbs">
<li><a href="contents.html">Docs</a> »</li>
<li>Development Guide</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/development_guide.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<img alt="Vespene Logo" class="align-right" src="_images/vespene_logo.png" />
<div class="section" id="development-guide">
<span id="id1"></span><h1>Development Guide<a class="headerlink" href="#development-guide" title="Permalink to this headline">¶</a></h1>
<p>If you are interested in developing on Vespene, excellent!</p>
<p>Maybe you just want to add a bug, or perhaps you want to add a plugin. If you’d like to share code back with us, that’s encouraged, but not required.
In this chapter, we’ll go over a little of everything.</p>
<p>If you are thinking about adding something new, see <a class="reference internal" href="resources.html#resources"><span class="std std-ref">Community Resources</span></a> for information about the discussion forum. Talking about ideas up front may
reduce duplicate efforts.</p>
<div class="section" id="architecture">
<h2>Architecture<a class="headerlink" href="#architecture" title="Permalink to this headline">¶</a></h2>
<p>There are two tiers to the application, technically - the web node and workers, but realistically every machine in a cluster is going to run the web node, and it is the number of workers that vary. Web nodes can run worker processes too, of course.</p>
<p>The web app uses Django (via Python 3) and talks to the PostgreSQL database. Workers use Django model code (but no web stack) and share the same database. Both are kicked off by supervisord, which in the production setup, is in turn launched by systemd (or whatever is the init system for the OS).</p>
</div>
<div class="section" id="code-structure">
<h2>Code Structure<a class="headerlink" href="#code-structure" title="Permalink to this headline">¶</a></h2>
<p>Inside the main checkout there are several directories, the main one ‘vespene’ has some important subdirectories to understand:</p>
<ul class="simple">
<li>common - code used by both the worker and web frontend (aka “manager”)</li>
<li>config - default settings to be used when there are no overriding settings in /etc/vespene/settings.d</li>
<li>jinja2 - templates used by the frontend</li>
<li>management - this is a django-standard directory for extensions to the ‘manage.py’ command line</li>
<li>manager - code that is specific to the web frontend and that does not run on the workers.</li>
<li>models - all ORM models with associated business logic</li>
<li>plugins - see :ref: <cite>plugins</cite>, these may be executed by the manager or worker depending on the type of plugin</li>
<li>static - random assets such as javascript support libraries and graphics.</li>
<li>views - code supporting the manager web UI. Business logic should go in ‘model’ objects or ‘manager’ instead (also plugins).</li>
<li>workers - logic that workers run to actually execute the builds. Can call into plugins, model, and common code.</li>
</ul>
<p>There are also some top level files:</p>
<ul class="simple">
<li>admin.py - entry point for the /admin Django admin interface</li>
<li>settings.py - entry point for settings, but mostly stored in config/</li>
<li>wsgi.py - entry point for gunicorn, our webserver</li>
</ul>
</div>
<div class="section" id="code-conventions">
<h2>Code Conventions<a class="headerlink" href="#code-conventions" title="Permalink to this headline">¶</a></h2>
<p>A few small rules:</p>
<ul class="simple">
<li>Mostly, code for maintaince so that the largest number of people possibly can understand it</li>
<li>Four space indent in Python is expected</li>
<li>PEP8 limits readability (whoa, controversy!), please do not submit PEP8 or whitespace corrections (sorry!) as these break attribution tracking (aka “git blame”).</li>
<li>Make sure python source files have the same license header as the other python files</li>
<li>Explain all files at the top of the files</li>
<li>Make decent use of docstring comments throughout files</li>
<li>Javascript should avoid using clippy.js - in general, limit javascript, we don’t want to require people to know a lot of it.</li>
</ul>
</div>
<div class="section" id="javascript-static-file-changes">
<h2>Javascript & Static File Changes<a class="headerlink" href="#javascript-static-file-changes" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li>application specific javascript is in vespene.js - there isn’t a lot</li>
<li>Right now there is a very small amount of javascript because the UI is still form based. This may not stay that way but was good for a start.</li>
<li>Be sure to run “python manage.py collectstatic” or changes you make in your editor won’t be served up by Django</li>
<li>All dependencies must be vendored - we want to enable Vespene to work on closed networks - so no CDNs.</li>
<li>We may have this minified better in the future.</li>
</ul>
</div>
<div class="section" id="database-changes">
<h2>Database Changes<a class="headerlink" href="#database-changes" title="Permalink to this headline">¶</a></h2>
<p>We must always think of existing users when making model changes, and be very careful when proposing model changes:</p>
<ul class="simple">
<li>“make migrations” must be run after making any database ORM changes, add any new files in “migrations/” to git with “git add <filename>” before sending a PR that adds a field.</li>
<li>Pay attention to any searches the application will use and add appropriate indexes in models.py when adding new fields when tables are going to be large</li>
<li>Minimize database migrations through list discussion, only one migration is allowed per feature, and we like to plan ahead and group them together as we can not easily consolidate them.</li>
<li>When adding fields to models, be sure to update forms.py</li>
<li>When adding new object types, add to models/, but also don’t forget the UI - views/, templates/theme.j2, and forms.py</li>
</ul>
</div>
<div class="section" id="adding-plugins">
<h2>Adding Plugins<a class="headerlink" href="#adding-plugins" title="Permalink to this headline">¶</a></h2>
<p>Look at the existing ‘plugins/’ folder and read <a class="reference internal" href="plugins.html#plugins"><span class="std std-ref">Plugins</span></a> for how they work.</p>
<p>To make a new plugin each class must be named Plugin. Nothing expensive (such as a network request) should be done in the constructor as the plugin may be instantiated quite frequently.</p>
<p>Plugins should be coded to be resilient to error and use meaningful logs and exceptions.</p>
<p>Plugins technically can call other plugins, but avoid doing so, as it might make the code confusing to trace.</p>
<p>Change settings.py to include the plugin if it is to be included by default, but likely we’ll need to describe the plugin on the plugins page either way if submitting it for inclusion in the main distribution. This is probably a discussion to be had on the pull request.</p>
<p>If in doubt ask questions on the <a class="reference external" href="https://talk.vespene.io">forum</a>, we’d be happy to help.</p>
</div>
<div class="section" id="dynamic-imports">
<h2>Dynamic Imports<a class="headerlink" href="#dynamic-imports" title="Permalink to this headline">¶</a></h2>
<p>If a plugin uses a library that not all users will be using, it can be dynamically imported like so:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">HAS_FOOLIB</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">try</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">some_dependency</span>
<span class="n">HAS_FOOLIB</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="k">pass</span>
</pre></div>
</div>
<p>And in the code for the plugin:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="k">if</span> <span class="ow">not</span> <span class="n">HAS_FOOLIB</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s2">"missing dependency: pip install foolib"</span><span class="p">)</span>
</pre></div>
</div>
<p>For instance, we would probably include a depdency for Slack or Hipchat, but would be unlikely to include
a dependency for a lesser used feature like Jabber.</p>
</div>
<div class="section" id="debugging-tips">
<h2>Debugging Tips<a class="headerlink" href="#debugging-tips" title="Permalink to this headline">¶</a></h2>
<p>This is all pretty standard web stack stuff, but mostly it helps to make sure you are seeing all the program output.</p>
<p>It’s easiest if you run the webserver from “make run” and the worker from commands like “ssh-agent manage.py worker <queue-name>” so you can see
standard output.</p>
<p>If you run things through the stock supervisor configuration, standard output will get sent to /var/log/vespene. Alternatively, you could
reconfigure supervisor.</p>
<p>Sorry, there are no major debugger tips here, but whatever you are used to will work fine.</p>
</div>
</div>
</div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
<a href="faq.html" class="btn btn-neutral float-right" title="FAQ/Troubleshooting" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
<a href="development_setup.html" class="btn btn-neutral" title="Development Setup" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
<hr/>
<div role="contentinfo">
<p>
© Copyright 2018, Michael DeHaan LLC.
</p>
</div>
Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT:'./',
VERSION:'',
LANGUAGE:'None',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/js/theme.js"></script>
<script type="text/javascript">
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
<style>
.wy-side-nav-search, .wy-nav-top {
background: #444444;
}
.wy-nav-side {
background: #444444;
}
</style>
</body>
</html>