Planetary Developer Docs: Drupal by Example

Joe Corneli edited this page Sep 30, 2013 · 6 revisions

Intro

These notes give an overview of what we’ve done so far in the Planetary project -- and more importantly, they will guide you through installation, debugging, writing your first module. They will grow with time to include basic coverage of the tricky Drupal bits like hooks, themes, how to use jQuery, etc.

Overview: Current Planetary Features (July, 2013)

Questions, requests, profile, private message, tags/sections, scoring, notices, user “ads”, orphanage, formatting, speed, backlinks, groups, editing UI...

Motivation to keep working!

See Planetary Use Cases and User Stories

Why using Drupal was a good idea

Views! We need a feed of problems without solutions, solutions without reviews, etc., and these are extremely easy to make.

Why using virtuoso was also a good idea

What links here? (Ask Virtuoso.)

Comparison with PlanetMath's legacy system (Noösphere)

It was a relatively quick change to Noosphere 1.5 to add problems and solutions. However, Noösphere is as flexible about manipulations of content or adding new features.

Issue reporting: a great first step!

  • Use the Planetary issue tracker for generic Planetary issues (can't go wrong). This includes LaTeXML customizations that are relevant only to PlanetMath or other Planetary deployments;

  • Use the LaTeXML Trac for LaTeXML-specific issues that show up in a Planetary instance, but that are ultimately relevant to other LaTeXML installations (account needed);

  • Use the PlanetMath issue tracker for PlanetMath organizational, strategic, and documentation issues;

  • Use the KWARC Planetary Trac instance for issues related to KWARC research prototypes that use Planetary (account needed);

  • Use the JOBAD issue tracker for most issues related to Javascript;

Adopting an issue

Check out the Hotfix and Code Cleanup milestones. There are lots of small things to improve or investigate as a way to learn.

Places to ask questions

Any questions about Planetary can go into the Planetary dev mailing list. We've also resolved to maintain a presence in the PlanetMath IRC channel, server irc.endernet.org, channel #planetmath.

You'll also want an account on drupal.org ("d.o.") and some acquaintance with the way they manage issue queues (e.g. make sure you mark "support requests" as such).

You can also ask generic Drupal questions on drupal.stackexchange.com. It's a great place to ask specific technical questions, especially if you're not getting help in the freenode #drupal-support IRC channel.

Tools

Drush

In general, Drush is awesome - check out this guide for some further details.

Note that you can do more with drush than you might have thought -- for example, there's no drush command to rebuild permissions, but you can just do drush php-eval 'node_access_rebuild();'.

Key points: use -v for verbose output and -v -d for even more verbose output.

Git

If you feel like you need to be a Github power user for a period of time, you can download and install GHI, which provides a CLI for Github.

Installation

The README contains step by step instructions for installing Drupal and other components.

You don't want to have to click through the UI every time you install, so you might want to get in the habit of doing this instead:

drush site-install [profile-name] --account-name=[admin_UN] --account-pass=[admin_PW] \
 --db-url=mysql://[db_user]:[db_pass]@localhost/[db_name] --site-name=Example

[profile-name] will be, for example, planetmath.

Getting involved - and debugging code. Use the source! -- you'll want these php commands:

  • dd - you can use this almost everywhere (prints things to /tmp/drupal_debug.txt)
  • dpm - useful for investigating complicated objects, but doesn't work in every instance
  • dpm(debug_backtrace()); - useful for figuring out how you get into a given block of code
  • You can use var_dump if dpm doesn't work.
  • Another nice trick is print_r($array,TRUE) which gives you something you can sorta copy and paste into code.
  • dd(array_keys($content)); is a nice way to get a succinct overview of the contents of an array, e.g. when you're rendering individual elements of an array in some template.
  • dpm(unserialize(...)) to make a readable version of serialized contents stored in MySQL

If you want to use dd in combination with drush, see this answer on SXE. You can also use drush_print (within Drush scripts).

Interactive Debugger!

If you're actively developing Drupal modules, it's helpful to have an interactive debugger. (Note: this section is strictly optional, and only of interest to developers.)

Like Drupal, Drush has its own sub-projects. To give yourself a comfortable interactive debugging environment, first download phpsh, and follow the provided installation instructions, minimally:

git clone git://github.com/facebook/phpsh.git
cd phpsh
python setup.py build
sudo python setup.py install

There's some preliminary drush integration but it only works so-so. (And it does NOT work well at all if you have Xdebug turned on.)

To get what's there, first install the Drush compatibility layer:

cd ~/.drush
git clone --branch master http://git.drupal.org/project/phpsh.git

Subsequently, you can run:

$ phpsh
php> r ~/.drush/phpsh/includes/console_7.inc
php> node_load(...);

If you apt-get install exuberant-ctags, you can then use drush phpsh-ctags to build an index of documentation, which you can then read interactively.

Writing your first module

Writing Drupal modules is a little bit confusing until you know what you're doing!

It might be helpful to share my own first module which demos a way to query an RDF triple store from within Drupal. It also shows how to use Drupal's "menu system" (see further notes below).

A couple additional modules that make good examples:

  • The correction module is by Alex Dumitru.
  • The problem module is by me, modeled after Alex's code. (A good example of how to adapt an existing module that does "almost" what you want.)

Some more detailed point-by-point walkthroughs will be gathered below.

Menus

Wildcards when defining a menu can be a little confusing unless it's spelled out.

Consider the path foo/bar/baz -- here bar occupies place 1. Thus, for example, we define the following menu item:

  $items['problems/%/attach'] = array(
      'title' => 'Attach problem',
      'type' => MENU_CALLBACK,
      'page callback' => 'problem_attach_problem',
      'page arguments' => array(1),
      'access arguments' => array('create correction content')
  );

Similarly, baz occupies place 2, and so we might define a more complicated menu item with multiple wildcards as follows:

  $items['problems/confirm/%/%'] = array(
      'title' => 'Confirm attachment',
      'type' => MENU_CALLBACK,
      'page callback' => 'problem_confirm_attachment',
      'page arguments' => array(2,3),//'nid_of_article','nid_of_problem'),
      'access arguments' => array('create correction content')
  );

In short, count the number of slashes, not the number of wildcards.

Drupal by Example

This is a rough outline of issues that came up working on a couple of modules, it needs a lot of work to be useful. The plan is to turn it into a quickstart guide to working with D7 and Planetary. It will need some rewriting

The installation profile

Features need to be loaded later in the sequence.

Selectively disabling some of the modules should help make the profile work better for dev work, i.e. faster.

  • Blocks
  • Config for pathautho
  • Figure out what the problem is with OG configuration in the profile.

Backups

drush ard all,default --destination=../Backups/archive-name.tar : scp that to disk from time to time (see: cron integration).

Restore via:

drush arr ../Backups/archive-name.tar

Pushing source code

Load the latest code onto Github.

find . -mtime -10 or N instead of 10. Copy files over.

Updating the core from time to time

...

Apply patches (if needed)

...

Note that applying patches to Drupal core is "taboo" and should be avoided, because it makes updating the site difficult. On the other hand, getting a given patch into D7 is not particularly easy. Typically the patches need to be applied to D8 and then backported.

Integrating with cron

...

Rebuild site content after importing

We have a drush script called rebuild.drush for that.

Getting URL aliasing sorted out

...

Integrating with a Version Control System

...

Implementing "watches" and notifications

Notify when:

  • an article is edited, a forum is updated
  • correction accepted/rejected
  • a solution is checked, a request is filled, orphaning is pending

Use: _watcher_user_set_watching_mode

Setting up fields

How do we know which collaborations are “public” and which are “private”?

Hooks. Example: planetmath_edit_article.

Canonical name shouldn’t be made editable to the user. There shouldn’t be spaces at the beginning or end of a title. We needed to make some things disappear from the editing UI. We need to deal with user preambles.

Module interconnections. Example: the interaction block .

We need some interactions for users and authors, and we want them to appear in a consistent way.

Do we need things like “unattach problem from this article”? How about “unattach correction”? The workflows are slightly different, that’s for sure. Corrections need an “accept” logic, whereas “reviews” do not. Note, this is similar to the question about reviewing articles on their way into the encyclopedia (see above).

Working with external programs: Example: SPARQL and metadata

More metadata: Example: Tracking user behavior

Building a custom datatype: Example: Collections (or “Math playlists”)

“Add to playlist” link (like on Youtube).

Suitable link to (my) playlists

Re-order items in a playlist, get the metadata/paradata display working: collections can show tasks, and “my progress”.

More raw notes to filter into the above

Things link like:

$A\rightarrow \{C,P\} S R$.

Add certification field to reviews: done

Show connections appropriately Get list of problems without solutions

Move the links at the bottom of the page into an “interact” box at the right.

“Revisions” to problems need to show up

Add back: sparql and sparql_registry

PyRDFa module should clean up after itself!

Maybe we should ship the code with ARC2 lib installed? Actually that doesn’t seem to work.

A virtuoso API. Maybe emphasize adding RDFa more?

\msc{47A32}
\author{jac}
\title{Happy Theorem}

Get metadata like this into the XHTML, extract it, put it into the store (all done: still need to check the deletion case however!).

We needed to install and populate the triple store with classification data. We didn’t find the famous sparql_views module to be particularly useful for our needs, maybe we just didn’t understand it?

With new articles: hook_node_insert (populate them with a suitable preamble) when updating: hook_node_presave.

What happens if the user has deleted some metadata fields or put in illegal values? (Well, we don’t let the user edit the TeX directly, we just take what they put into the form... but nevertheless, we might need to do some checking for }, for example, since that could screw up the tex processor).

Later: svn commit myarticle.tex -m "Bla" (and extract metadata as part of the post-commit).

We should also be updating the triple store information when the user transfers ownership.

Rather than storing the metadata field “within” the user-editable preamble field, we just decided to go ahead and create an extra textarea as part of the latex_field.

If we want to store triples like “16puzzle is related to 17 puzzle” then we will have some work to do.

Triple or other way to find things:

MSC Categories $\rightarrow$ Articles$ \rightarrow$ Problems

Problems could also connect to other problems (hasAssociatedProblem).

We could do some kind of “fuzzy annotations” but that might be expensive.

We’d like to be able to do: “Give me problems about games that have solutions”. (This might be a place for SPARQL views?)

Deyan: design an RDFa binding for something like \addproblem{SquarePuzzleProblem}.

We need a way to get the sparql store location set. (Actually we can just stick this in msc_browser).

Hook so that RDFa extractor runs on “save” event. Replace the old data with new data. Tricky, because we need to get it at the right point in time.

Extra bonus: nice displays of query results.

(Note: caching the MSC information would provide a nice speed up...)

\pmproposedsoln{SquarePuzzleSoln1}

Add a solution. \pmsolves{SquarePuzzle}.

Press save.

Instead of using a triple-based solution, we ended up just maintaining a table of article nids and associated problem nids.

Problem 1: use a flag and node_load to do the metadata processing?

This is basically an example of the “metadata” topic discussed in the previous section – though the term “paradata” has been advanced to describe data about use (as opposed to data about content), and I tend to prefer this term.

In particular: if you’re not logged in, you don’t get any personal metadata!

Considering using Virtuoso or Drupal for activity data...

Export current config for userpoints. userpoints_nc?

We need a “scoresheet” extracted from the user score information.

Users and workflow

Description logics. Users. Can we say “What a user knows about”? Can we say who is an “expert”?

Is there a “review” step on the way to publishing something in the encyclopedia?

Cylinder sets... can show us if two people are working on “similar topics” at a given point in time. Then we might want to put them together to form a little ad hoc study group.

Groups and permissions

Every article is a group? Not any more.

Coauthor groups

Groups and permissions - Note: for the most part, groups have been our method for handling permissions (describe this)

They are: site-wide, user-specific, generated partly on the fly.

Apply to become a co-author?

Designate an existing team as a buddy list?

Workflow related to teams: Can Lucas have a list of algebra articles by Joe in his team (that he can’t edit, until he invites Joe, who can then transfer some subset of the articles, etc)?

(Is my workplan article publicly editable? Maybe I need to share it with specific co-authors. Note that there is still that “group loophole” to deal with.)

Information about buddy lists had to be recorded in a couple of places (the user object, and the group system)...

Joining a group: Request, confirm, add? Request, Add?

Different places where we may want to have different levels of permissions, questioning, triple handshakes, etc

Button/block to transfer articles: done

Text on nodes should say “owner” not “submitted by” or whatever.

The “groups audience” should be shown on the article.’

Need: group permissions to be squared away

Groups policy: 1. Coauthor group of article A – all coauthors of A can edit A. 2. Team – all members of a team can edit all articles shared by the team 3. Buddy list of U – all members can edit all items created by U

Groups: who’s in them? Search across them? Search within them? Associate with content.

Group stuff: information needs to show up on user pages, group pages, articles, the group list...

Adding content should be easy...

“Forums intersect group” - to show all postings from people in e.g. a course

How will people coordinate work?

Can we catch up with what people (e.g. in a group) are doing by using a shared activity stream?

Migration

Migration-related TODO items: migrate requests, thinking of them as a type of question

Comments: Node, OP, reply to etc. all had to be sorted out

Errors mid-migration: where do these come from?

Does Correction import work?

There should be a fairly simple way to overwrite the Drupal-generated score with the old score. And maybe some way to replay the “activities” that went into making that score (though that seems a bit doubtful).

Collaboration style

  • Time, presence, money, emotions, taking care, healthy boundaries...
  • Feeling like it’s a chore vs feeling like it’s a creative process...
  • “fractal knowledge”...

Drupal internals...

Where is info about the installed modules stored in Drupal’s database?

Example of development cycle: Finishing up the PlanetMath alpha

This was the kind of tasklist that confronted me:

  • Complete the different functionality around problem solving
  • Check into github those things which need to be there
  • Import content, using a recent database dump
  • Check that all the needed interactions are there
  • Look at the list and refine the list of things that happen
  • Collections (browsing the list of all problems, for example, is too much).