Skip to content

Commit 4adf559

Browse files
author
epriestley
committed
Port Diviner Core to Phabricator
Summary: This implements most/all of the difficult parts of Diviner on top of Phabricator instead of as standalone components. See T988. In particular, here are the things I want to fix: **Performance** The Diviner parser works in two stages. The first stage breaks source files into "Atoms". The second stage renders atoms into a display format (e.g., HTML). Diviner currently has a good caching story on the first step of the pipeline, but zero caching in the second step. This means it's very slow, even for a fairly small project like Phabricator. We must re-render every piece of documentation every time, instead of only changed documentation. Most of this diff concerns itself with addressing this problem. There's a fairly large explanatory comment about it, but the trickiest part is that when an atom changes, other atoms (defined in other places) may also change -- for example, if `class B extends A`, editing A should dirty B, even if B is in an entirely different file. We perform analysis in two stages to propagate these changes: first detecting direct changes, then detecting indirect changes. This isn't completely implemented -- we need to propagate 'extends' through more levels -- but I believe it's structurally correct and good enough until we actually document classes. **Inheritance** Diviner currently has a very weak story on inheritance. I want to inherit a lot more metas/docs. If an interface documents a method, we should just pull that documentation in to every implementation by default (implementations can still override it if they want). It can be shown in grey or something, but it should be desirable and correct to omit documentation of a method implementation when you are implementing a parent. Similarly, I want to pull in inherited methods and @tasks and such. This diff sets up for that, by formalizing "extends" relationships between atoms. **Overspecialization** Diviner currently specializes atoms (FileAtom, FunctionAtom, ClassAtom, etc.). This is pretty much not useful, because Atomizers (which produce the atoms) need to be highly specialized, and Renderers/Publishers (which consume the atoms) also need to be highly specialized. Nothing interesting actually lives in the atom specializations, and we don't benefit from having them -- it just costs us generality in storage/caches for them. In the new code, I've used a single Atom class to represent any type of atom. **URIs** We have fairly hideous URIs right now, which are very cumbersome For in-app doc links, I want to provide nice URIs ("/h/notfications" or similar) which are stable redirects, and probably add remarkup for it: !{notifications} or similar. This diff isn't related to that since it's too premature. **Search** Once we have a database generation target, we can index the documentation. **Design** Chad has some nice mocks. Test Plan: Ran `bin/diviner generate`, `bin/diviner generate --clean`. Saw appropriate graph propagation after edits. This diff doesn't do anything very useful yet. Reviewers: btrahan, vrana Reviewed By: btrahan CC: aran Maniphest Tasks: T988 Differential Revision: https://secure.phabricator.com/D4340
1 parent f7939b9 commit 4adf559

File tree

12 files changed

+1260
-0
lines changed

12 files changed

+1260
-0
lines changed

‎bin/diviner

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../scripts/diviner/diviner.php

‎scripts/diviner/diviner.php

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env php
2+
<?php
3+
4+
$root = dirname(dirname(dirname(__FILE__)));
5+
require_once $root.'/scripts/__init_script__.php';
6+
7+
$args = new PhutilArgumentParser($argv);
8+
9+
$args->setTagline('documentation generator');
10+
$args->setSynopsis(<<<EOHELP
11+
**diviner** __command__ [__options__]
12+
Generate documentation.
13+
EOHELP
14+
);
15+
$args->parseStandardArguments();
16+
17+
$args->parseWorkflows(
18+
array(
19+
new DivinerGenerateWorkflow(),
20+
new DivinerAtomizeWorkflow(),
21+
new PhutilHelpArgumentWorkflow(),
22+
));

‎src/__phutil_library_map__.php

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -422,7 +422,16 @@
422422
'DiffusionTagListView' => 'applications/diffusion/view/DiffusionTagListView.php',
423423
'DiffusionURITestCase' => 'applications/diffusion/request/__tests__/DiffusionURITestCase.php',
424424
'DiffusionView' => 'applications/diffusion/view/DiffusionView.php',
425+
'DivinerArticleAtomizer' => 'applications/diviner/atomizer/DivinerArticleAtomizer.php',
426+
'DivinerAtom' => 'applications/diviner/atom/DivinerAtom.php',
427+
'DivinerAtomCache' => 'applications/diviner/cache/DivinerAtomCache.php',
428+
'DivinerAtomRef' => 'applications/diviner/atom/DivinerAtomRef.php',
429+
'DivinerAtomizeWorkflow' => 'applications/diviner/workflow/DivinerAtomizeWorkflow.php',
430+
'DivinerAtomizer' => 'applications/diviner/atomizer/DivinerAtomizer.php',
431+
'DivinerFileAtomizer' => 'applications/diviner/atomizer/DivinerFileAtomizer.php',
432+
'DivinerGenerateWorkflow' => 'applications/diviner/workflow/DivinerGenerateWorkflow.php',
425433
'DivinerListController' => 'applications/diviner/controller/DivinerListController.php',
434+
'DivinerWorkflow' => 'applications/diviner/workflow/DivinerWorkflow.php',
426435
'DrydockAllocatorWorker' => 'applications/drydock/worker/DrydockAllocatorWorker.php',
427436
'DrydockApacheWebrootInterface' => 'applications/drydock/interface/webroot/DrydockApacheWebrootInterface.php',
428437
'DrydockBlueprint' => 'applications/drydock/blueprint/DrydockBlueprint.php',
@@ -1780,7 +1789,12 @@
17801789
'DiffusionTagListView' => 'DiffusionView',
17811790
'DiffusionURITestCase' => 'ArcanistPhutilTestCase',
17821791
'DiffusionView' => 'AphrontView',
1792+
'DivinerArticleAtomizer' => 'DivinerAtomizer',
1793+
'DivinerAtomizeWorkflow' => 'DivinerWorkflow',
1794+
'DivinerFileAtomizer' => 'DivinerAtomizer',
1795+
'DivinerGenerateWorkflow' => 'DivinerWorkflow',
17831796
'DivinerListController' => 'PhabricatorController',
1797+
'DivinerWorkflow' => 'PhutilArgumentWorkflow',
17841798
'DrydockAllocatorWorker' => 'PhabricatorWorker',
17851799
'DrydockApacheWebrootInterface' => 'DrydockWebrootInterface',
17861800
'DrydockCommandInterface' => 'DrydockInterface',

‎src/applications/diviner/atom/DivinerAtom.php

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
<?php
2+
3+
final class DivinerAtom {
4+
5+
const TYPE_FILE = 'file';
6+
const TYPE_ARTICLE = 'article';
7+
8+
private $type;
9+
private $name;
10+
private $file;
11+
private $line;
12+
private $hash;
13+
private $contentRaw;
14+
private $length;
15+
private $language;
16+
private $docblockRaw;
17+
private $docblockText;
18+
private $docblockMeta;
19+
private $warnings = array();
20+
private $parentHash;
21+
private $childHashes = array();
22+
private $context;
23+
private $extends = array();
24+
private $links = array();
25+
private $project;
26+
27+
public function setProject($project) {
28+
$this->project = $project;
29+
return $this;
30+
}
31+
32+
public function getProject() {
33+
return $this->project;
34+
}
35+
36+
public function setContext($context) {
37+
$this->context = $context;
38+
return $this;
39+
}
40+
41+
public function getContext() {
42+
return $this->context;
43+
}
44+
45+
public static function getAtomSerializationVersion() {
46+
return 1;
47+
}
48+
49+
public function addWarning($warning) {
50+
$this->warnings[] = $warning;
51+
return $this;
52+
}
53+
54+
public function getWarnings() {
55+
return $this->warnings;
56+
}
57+
58+
public function setDocblockRaw($docblock_raw) {
59+
$this->docblockRaw = $docblock_raw;
60+
61+
$parser = new PhutilDocblockParser();
62+
list($text, $meta) = $parser->parse($docblock_raw);
63+
$this->docblockText = $text;
64+
$this->docblockMeta = $meta;
65+
66+
return $this;
67+
}
68+
69+
public function getDocblockRaw() {
70+
return $this->docblockRaw;
71+
}
72+
73+
public function getDocblockText() {
74+
if ($this->docblockText === null) {
75+
throw new Exception("Call setDocblockRaw() before getDocblockText()!");
76+
}
77+
return $this->docblockText;
78+
}
79+
80+
public function getDocblockMeta() {
81+
if ($this->docblockMeta === null) {
82+
throw new Exception("Call setDocblockRaw() before getDocblockMeta()!");
83+
}
84+
return $this->docblockMeta;
85+
}
86+
87+
public function setType($type) {
88+
$this->type = $type;
89+
return $this;
90+
}
91+
92+
public function getType() {
93+
return $this->type;
94+
}
95+
96+
public function setName($name) {
97+
$this->name = $name;
98+
return $this;
99+
}
100+
101+
public function getName() {
102+
return $this->name;
103+
}
104+
105+
public function setFile($file) {
106+
$this->file = $file;
107+
return $this;
108+
}
109+
110+
public function getFile() {
111+
return $this->file;
112+
}
113+
114+
public function setLine($line) {
115+
$this->line = $line;
116+
return $this;
117+
}
118+
119+
public function getLine() {
120+
return $this->line;
121+
}
122+
123+
public function setContentRaw($content_raw) {
124+
$this->contentRaw = $content_raw;
125+
return $this;
126+
}
127+
128+
public function getContentRaw() {
129+
return $this->contentRaw;
130+
}
131+
132+
public function setHash($hash) {
133+
$this->hash = $hash;
134+
return $this;
135+
}
136+
137+
public function addLink(DivinerAtomRef $ref) {
138+
$this->links[] = $ref;
139+
return $this;
140+
}
141+
142+
public function addExtends(DivinerAtomRef $ref) {
143+
$this->extends[] = $ref;
144+
return $this;
145+
}
146+
147+
public function getLinkDictionaries() {
148+
return mpull($this->links, 'toDictionary');
149+
}
150+
151+
public function getExtendsDictionaries() {
152+
return mpull($this->extends, 'toDictionary');
153+
}
154+
155+
public function getHash() {
156+
if ($this->hash) {
157+
return $this->hash;
158+
}
159+
160+
$parts = array(
161+
$this->getType(),
162+
$this->getName(),
163+
$this->getFile(),
164+
$this->getLine(),
165+
$this->getLength(),
166+
$this->getLanguage(),
167+
$this->getContentRaw(),
168+
$this->getDocblockRaw(),
169+
mpull($this->extends, 'toHash'),
170+
mpull($this->links, 'toHash'),
171+
);
172+
173+
return md5(serialize($parts)).'N';
174+
}
175+
176+
public function setLength($length) {
177+
$this->length = $length;
178+
return $this;
179+
}
180+
181+
public function getLength() {
182+
return $this->length;
183+
}
184+
185+
public function setLanguage($language) {
186+
$this->language = $language;
187+
return $this;
188+
}
189+
190+
public function getLanguage() {
191+
return $this->language;
192+
}
193+
194+
public function addChildHash($child_hash) {
195+
$this->childHashes[] = $child_hash;
196+
return $this;
197+
}
198+
199+
public function getChildHashes() {
200+
return $this->childHashes;
201+
}
202+
203+
public function setParentHash($parent_hash) {
204+
if ($this->parentHash) {
205+
throw new Exception("Atom already has a parent!");
206+
}
207+
$this->parentHash = $parent_hash;
208+
return $this;
209+
}
210+
211+
public function getParentHash() {
212+
return $this->parentHash;
213+
}
214+
215+
public function addChild(DivinerAtom $atom) {
216+
$atom->setParentHash($this->getHash());
217+
$this->addChildHash($atom->getHash());
218+
return $this;
219+
}
220+
221+
public function getURI() {
222+
$parts = array();
223+
$parts[] = phutil_escape_uri_path_component($this->getType());
224+
if ($this->getContext()) {
225+
$parts[] = phutil_escape_uri_path_component($this->getContext());
226+
}
227+
$parts[] = phutil_escape_uri_path_component($this->getName());
228+
$parts[] = null;
229+
return implode('/', $parts);
230+
}
231+
232+
233+
public function toDictionary() {
234+
// NOTE: If you change this format, bump the format version in
235+
// getAtomSerializationVersion().
236+
237+
return array(
238+
'type' => $this->getType(),
239+
'name' => $this->getName(),
240+
'file' => $this->getFile(),
241+
'line' => $this->getLine(),
242+
'hash' => $this->getHash(),
243+
'uri' => $this->getURI(),
244+
'length' => $this->getLength(),
245+
'context' => $this->getContext(),
246+
'language' => $this->getLanguage(),
247+
'docblockRaw' => $this->getDocblockRaw(),
248+
'warnings' => $this->getWarnings(),
249+
'parentHash' => $this->getParentHash(),
250+
'childHashes' => $this->getChildHashes(),
251+
'extends' => $this->getExtendsDictionaries(),
252+
'links' => $this->getLinkDictionaries(),
253+
'ref' => $this->getRef()->toDictionary(),
254+
);
255+
}
256+
257+
public function getRef() {
258+
return id(new DivinerAtomRef())
259+
->setProject($this->getProject())
260+
->setContext($this->getContext())
261+
->setType($this->getType())
262+
->setName($this->getName());
263+
}
264+
265+
public static function newFromDictionary(array $dictionary) {
266+
$atom = id(new DivinerAtom())
267+
->setType(idx($dictionary, 'type'))
268+
->setName(idx($dictionary, 'name'))
269+
->setFile(idx($dictionary, 'file'))
270+
->setLine(idx($dictionary, 'line'))
271+
->setHash(idx($dictionary, 'hash'))
272+
->setLength(idx($dictionary, 'length'))
273+
->setContext(idx($dictionary, 'context'))
274+
->setLanguage(idx($dictionary, 'language'))
275+
->setParentHash(idx($dictionary, 'parentHash'))
276+
->setDocblockRaw(idx($dictionary, 'docblockRaw'));
277+
278+
foreach (idx($dictionary, 'warnings', array()) as $warning) {
279+
$atom->addWarning($warning);
280+
}
281+
282+
foreach (idx($dictionary, 'childHashes', array()) as $child) {
283+
$atom->addChildHash($child);
284+
}
285+
286+
return $atom;
287+
}
288+
289+
}

0 commit comments

Comments
 (0)