Permalink
Browse files

Port Diviner Core to Phabricator

Summary:
This implements most/all of the difficult parts of Diviner on top of Phabricator instead of as standalone components. See T988. In particular, here are the things I want to fix:

**Performance** The Diviner parser works in two stages. The first stage breaks source files into "Atoms". The second stage renders atoms into a display format (e.g., HTML). Diviner currently has a good caching story on the first step of the pipeline, but zero caching in the second step. This means it's very slow, even for a fairly small project like Phabricator. We must re-render every piece of documentation every time, instead of only changed documentation. Most of this diff concerns itself with addressing this problem. There's a fairly large explanatory comment about it, but the trickiest part is that when an atom changes, other atoms (defined in other places) may also change -- for example, if `class B extends A`, editing A should dirty B, even if B is in an entirely different file. We perform analysis in two stages to propagate these changes: first detecting direct changes, then detecting indirect changes. This isn't completely implemented -- we need to propagate 'extends' through more levels -- but I believe it's structurally correct and good enough until we actually document classes.

**Inheritance** Diviner currently has a very weak story on inheritance. I want to inherit a lot more metas/docs. If an interface documents a method, we should just pull that documentation in to every implementation by default (implementations can still override it if they want). It can be shown in grey or something, but it should be desirable and correct to omit documentation of a method implementation when you are implementing a parent. Similarly, I want to pull in inherited methods and @tasks and such. This diff sets up for that, by formalizing "extends" relationships between atoms.

**Overspecialization** Diviner currently specializes atoms (FileAtom, FunctionAtom, ClassAtom, etc.). This is pretty much not useful, because Atomizers (which produce the atoms) need to be highly specialized, and Renderers/Publishers (which consume the atoms) also need to be highly specialized. Nothing interesting actually lives in the atom specializations, and we don't benefit from having them -- it just costs us generality in storage/caches for them. In the new code, I've used a single Atom class to represent any type of atom.

**URIs** We have fairly hideous URIs right now, which are very cumbersome  For in-app doc links, I want to provide nice URIs ("/h/notfications" or similar) which are stable redirects, and probably add remarkup for it: !{notifications} or similar. This diff isn't related to that since it's too premature.

**Search** Once we have a database generation target, we can index the documentation.

**Design** Chad has some nice mocks.

Test Plan: Ran `bin/diviner generate`, `bin/diviner generate --clean`. Saw appropriate graph propagation after edits. This diff doesn't do anything very useful yet.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T988

Differential Revision: https://secure.phabricator.com/D4340
  • Loading branch information...
1 parent f7939b9 commit 4adf55919c755e301b515d6eeb618250840a55b6 @epriestley epriestley committed Jan 7, 2013
View
@@ -0,0 +1,22 @@
+#!/usr/bin/env php
+<?php
+
+$root = dirname(dirname(dirname(__FILE__)));
+require_once $root.'/scripts/__init_script__.php';
+
+$args = new PhutilArgumentParser($argv);
+
+$args->setTagline('documentation generator');
+$args->setSynopsis(<<<EOHELP
+**diviner** __command__ [__options__]
+ Generate documentation.
+EOHELP
+);
+$args->parseStandardArguments();
+
+$args->parseWorkflows(
+ array(
+ new DivinerGenerateWorkflow(),
+ new DivinerAtomizeWorkflow(),
+ new PhutilHelpArgumentWorkflow(),
+ ));
@@ -422,7 +422,16 @@
'DiffusionTagListView' => 'applications/diffusion/view/DiffusionTagListView.php',
'DiffusionURITestCase' => 'applications/diffusion/request/__tests__/DiffusionURITestCase.php',
'DiffusionView' => 'applications/diffusion/view/DiffusionView.php',
+ 'DivinerArticleAtomizer' => 'applications/diviner/atomizer/DivinerArticleAtomizer.php',
+ 'DivinerAtom' => 'applications/diviner/atom/DivinerAtom.php',
+ 'DivinerAtomCache' => 'applications/diviner/cache/DivinerAtomCache.php',
+ 'DivinerAtomRef' => 'applications/diviner/atom/DivinerAtomRef.php',
+ 'DivinerAtomizeWorkflow' => 'applications/diviner/workflow/DivinerAtomizeWorkflow.php',
+ 'DivinerAtomizer' => 'applications/diviner/atomizer/DivinerAtomizer.php',
+ 'DivinerFileAtomizer' => 'applications/diviner/atomizer/DivinerFileAtomizer.php',
+ 'DivinerGenerateWorkflow' => 'applications/diviner/workflow/DivinerGenerateWorkflow.php',
'DivinerListController' => 'applications/diviner/controller/DivinerListController.php',
+ 'DivinerWorkflow' => 'applications/diviner/workflow/DivinerWorkflow.php',
'DrydockAllocatorWorker' => 'applications/drydock/worker/DrydockAllocatorWorker.php',
'DrydockApacheWebrootInterface' => 'applications/drydock/interface/webroot/DrydockApacheWebrootInterface.php',
'DrydockBlueprint' => 'applications/drydock/blueprint/DrydockBlueprint.php',
@@ -1780,7 +1789,12 @@
'DiffusionTagListView' => 'DiffusionView',
'DiffusionURITestCase' => 'ArcanistPhutilTestCase',
'DiffusionView' => 'AphrontView',
+ 'DivinerArticleAtomizer' => 'DivinerAtomizer',
+ 'DivinerAtomizeWorkflow' => 'DivinerWorkflow',
+ 'DivinerFileAtomizer' => 'DivinerAtomizer',
+ 'DivinerGenerateWorkflow' => 'DivinerWorkflow',
'DivinerListController' => 'PhabricatorController',
+ 'DivinerWorkflow' => 'PhutilArgumentWorkflow',
'DrydockAllocatorWorker' => 'PhabricatorWorker',
'DrydockApacheWebrootInterface' => 'DrydockWebrootInterface',
'DrydockCommandInterface' => 'DrydockInterface',
@@ -0,0 +1,289 @@
+<?php
+
+final class DivinerAtom {
+
+ const TYPE_FILE = 'file';
+ const TYPE_ARTICLE = 'article';
+
+ private $type;
+ private $name;
+ private $file;
+ private $line;
+ private $hash;
+ private $contentRaw;
+ private $length;
+ private $language;
+ private $docblockRaw;
+ private $docblockText;
+ private $docblockMeta;
+ private $warnings = array();
+ private $parentHash;
+ private $childHashes = array();
+ private $context;
+ private $extends = array();
+ private $links = array();
+ private $project;
+
+ public function setProject($project) {
+ $this->project = $project;
+ return $this;
+ }
+
+ public function getProject() {
+ return $this->project;
+ }
+
+ public function setContext($context) {
+ $this->context = $context;
+ return $this;
+ }
+
+ public function getContext() {
+ return $this->context;
+ }
+
+ public static function getAtomSerializationVersion() {
+ return 1;
+ }
+
+ public function addWarning($warning) {
+ $this->warnings[] = $warning;
+ return $this;
+ }
+
+ public function getWarnings() {
+ return $this->warnings;
+ }
+
+ public function setDocblockRaw($docblock_raw) {
+ $this->docblockRaw = $docblock_raw;
+
+ $parser = new PhutilDocblockParser();
+ list($text, $meta) = $parser->parse($docblock_raw);
+ $this->docblockText = $text;
+ $this->docblockMeta = $meta;
+
+ return $this;
+ }
+
+ public function getDocblockRaw() {
+ return $this->docblockRaw;
+ }
+
+ public function getDocblockText() {
+ if ($this->docblockText === null) {
+ throw new Exception("Call setDocblockRaw() before getDocblockText()!");
+ }
+ return $this->docblockText;
+ }
+
+ public function getDocblockMeta() {
+ if ($this->docblockMeta === null) {
+ throw new Exception("Call setDocblockRaw() before getDocblockMeta()!");
+ }
+ return $this->docblockMeta;
+ }
+
+ public function setType($type) {
+ $this->type = $type;
+ return $this;
+ }
+
+ public function getType() {
+ return $this->type;
+ }
+
+ public function setName($name) {
+ $this->name = $name;
+ return $this;
+ }
+
+ public function getName() {
+ return $this->name;
+ }
+
+ public function setFile($file) {
+ $this->file = $file;
+ return $this;
+ }
+
+ public function getFile() {
+ return $this->file;
+ }
+
+ public function setLine($line) {
+ $this->line = $line;
+ return $this;
+ }
+
+ public function getLine() {
+ return $this->line;
+ }
+
+ public function setContentRaw($content_raw) {
+ $this->contentRaw = $content_raw;
+ return $this;
+ }
+
+ public function getContentRaw() {
+ return $this->contentRaw;
+ }
+
+ public function setHash($hash) {
+ $this->hash = $hash;
+ return $this;
+ }
+
+ public function addLink(DivinerAtomRef $ref) {
+ $this->links[] = $ref;
+ return $this;
+ }
+
+ public function addExtends(DivinerAtomRef $ref) {
+ $this->extends[] = $ref;
+ return $this;
+ }
+
+ public function getLinkDictionaries() {
+ return mpull($this->links, 'toDictionary');
+ }
+
+ public function getExtendsDictionaries() {
+ return mpull($this->extends, 'toDictionary');
+ }
+
+ public function getHash() {
+ if ($this->hash) {
+ return $this->hash;
+ }
+
+ $parts = array(
+ $this->getType(),
+ $this->getName(),
+ $this->getFile(),
+ $this->getLine(),
+ $this->getLength(),
+ $this->getLanguage(),
+ $this->getContentRaw(),
+ $this->getDocblockRaw(),
+ mpull($this->extends, 'toHash'),
+ mpull($this->links, 'toHash'),
+ );
+
+ return md5(serialize($parts)).'N';
+ }
+
+ public function setLength($length) {
+ $this->length = $length;
+ return $this;
+ }
+
+ public function getLength() {
+ return $this->length;
+ }
+
+ public function setLanguage($language) {
+ $this->language = $language;
+ return $this;
+ }
+
+ public function getLanguage() {
+ return $this->language;
+ }
+
+ public function addChildHash($child_hash) {
+ $this->childHashes[] = $child_hash;
+ return $this;
+ }
+
+ public function getChildHashes() {
+ return $this->childHashes;
+ }
+
+ public function setParentHash($parent_hash) {
+ if ($this->parentHash) {
+ throw new Exception("Atom already has a parent!");
+ }
+ $this->parentHash = $parent_hash;
+ return $this;
+ }
+
+ public function getParentHash() {
+ return $this->parentHash;
+ }
+
+ public function addChild(DivinerAtom $atom) {
+ $atom->setParentHash($this->getHash());
+ $this->addChildHash($atom->getHash());
+ return $this;
+ }
+
+ public function getURI() {
+ $parts = array();
+ $parts[] = phutil_escape_uri_path_component($this->getType());
+ if ($this->getContext()) {
+ $parts[] = phutil_escape_uri_path_component($this->getContext());
+ }
+ $parts[] = phutil_escape_uri_path_component($this->getName());
+ $parts[] = null;
+ return implode('/', $parts);
+ }
+
+
+ public function toDictionary() {
+ // NOTE: If you change this format, bump the format version in
+ // getAtomSerializationVersion().
+
+ return array(
+ 'type' => $this->getType(),
+ 'name' => $this->getName(),
+ 'file' => $this->getFile(),
+ 'line' => $this->getLine(),
+ 'hash' => $this->getHash(),
+ 'uri' => $this->getURI(),
+ 'length' => $this->getLength(),
+ 'context' => $this->getContext(),
+ 'language' => $this->getLanguage(),
+ 'docblockRaw' => $this->getDocblockRaw(),
+ 'warnings' => $this->getWarnings(),
+ 'parentHash' => $this->getParentHash(),
+ 'childHashes' => $this->getChildHashes(),
+ 'extends' => $this->getExtendsDictionaries(),
+ 'links' => $this->getLinkDictionaries(),
+ 'ref' => $this->getRef()->toDictionary(),
+ );
+ }
+
+ public function getRef() {
+ return id(new DivinerAtomRef())
+ ->setProject($this->getProject())
+ ->setContext($this->getContext())
+ ->setType($this->getType())
+ ->setName($this->getName());
+ }
+
+ public static function newFromDictionary(array $dictionary) {
+ $atom = id(new DivinerAtom())
+ ->setType(idx($dictionary, 'type'))
+ ->setName(idx($dictionary, 'name'))
+ ->setFile(idx($dictionary, 'file'))
+ ->setLine(idx($dictionary, 'line'))
+ ->setHash(idx($dictionary, 'hash'))
+ ->setLength(idx($dictionary, 'length'))
+ ->setContext(idx($dictionary, 'context'))
+ ->setLanguage(idx($dictionary, 'language'))
+ ->setParentHash(idx($dictionary, 'parentHash'))
+ ->setDocblockRaw(idx($dictionary, 'docblockRaw'));
+
+ foreach (idx($dictionary, 'warnings', array()) as $warning) {
+ $atom->addWarning($warning);
+ }
+
+ foreach (idx($dictionary, 'childHashes', array()) as $child) {
+ $atom->addChildHash($child);
+ }
+
+ return $atom;
+ }
+
+}
Oops, something went wrong.

0 comments on commit 4adf559

Please sign in to comment.