Skip to content


jaybonci edited this page Mar 20, 2012 · 5 revisions
Clone this wiki locally

Here's the generalized todo list. This is going to be getting a lot of the stuff out of my head, and we can talk about it.


  • Better offsite backup strategy

  • Varnish/NGINX caching for Guest User pages

  • DNS stability

Engine work

Engine changes:

  • Support for MySQL views: Currently we have two mysql views which exist in production. Work needs to be done to analyze how we use them, and they need to be created on bootstrap properly, so we need to fully support their creation as a nodetype

    • This work is completed in dev, but it needs to be integrated into production, in a somewhat lengthy catchup process.
    • Secondarily, bootstrap needs to handle this in the process after the database structures are inserted. This is because when we create the nodes in the database, we insert the fields across the columns, but do not actually call the maintenance types
  • Support for MySQL procedures: There are two procedures used in production that we need to support in the development environment and determine whether or not they are used.

    • Similar to the bootstrapping issue with MySQL views, special handling needs to take place to make it work on standup
  • Delta-encoded nodecache: A stopgap piece of work that I'm going to be bringing to a technical discussion on PerlMonks is the concept of having a delta encoded nodecache for static nodetypes. Most of our database traffic on Guest User pages is to the version table to check for the current version of a node. The static nodetypes do not change very much, and so when they do, we can insert a record into a table which will list the timestamp of the change, and the node_ids that are changed. Instead of checking the version on a static nodetype entry, we will go and get the last delta time, and if it is ahead of our last updated time, we'll get the list of nodes to tombstone.

  • Version table as HEAP. If I'm sure that we can support the memory usage needed for a HEAP type for the version table, I would do it. It's a much faster

  • Remove the node.lock column, pending work on InnoDB transactions. The lock column is a hack for updateNode() for a low-budget mutex. InnoDB has transactions, so we can remove that column and save data in the nodecache

  • Evaluate and remove the core column in node - We don't need the core column as it matches an outdated model that I think maps to the nodeball setup. Now that we are going to be operating out of git, then we can remove it and reduce the nodecache footprint

  • Remove the _ORIGINAL_VALUES update piece. Currently, we have a nodecache size issue caused by the _ORIGINAL_VALUES work. Basically, this is to create an update reference copy so that things do not get clobbered. There are two potential fixes to this:

    • Implement a per-page temporary nodecache where we clone items into it as a reference for updates. This also provides a secondary benefit of shielding against NodeCache poisoning by updates to nodes that are not tracked into the database, a shadow problem with our data model. We could also have a debug function which checks for poisoning before discarding the per-page cache.
  • Remove the sqlLog - We have development boxes, and we can turn mysql logging on if need be. This feature needs to be removed. It's only a memory sink.

  • Remove the nodeball type - We don't need nodeballs anymore. Themes descend from them, but it's pretty easy to untangle that.

  • Create a mechanism by which superdocs (and others) can redirect if permissions are denied or if it doesn't apply for a particular node. For instance, if Guest User/Anonymous Monk hits Message Inbox, they should be bounced to the login page, and the login page should bounce them back to the Message inbox. Right now there is boilerplate code inside of a pile of docs and types where it basically says: return "NO ACCESS FOR YOU" if $USER->{node_id} == $HTMLVARS{guest_user}; That's kind of ridiculous to have to put into every superdoc. It'd be better and more helpful to have access types associated with them. It might also make sense to supersede restricted, oppressor, and edev docs.

  • Make it so that any node can have settings. This going along with the next point, but since the actual vars hash needs to be unpacked before it's used, makes it no sense that joining nodes on the settings table when we might not be calling getVars on it. This is of particular use to the user nodetype, who is called a lot. We

    • This might need to change the way settings are handled in the NodeCache and isn't as simple a change as one might expect.
  • Make it so that settings are individually stored and accessible. Currently the pack and store of node settings makes it difficult to manage preferences across the code of the userbase. We need to have a vars param table that is where individual hash items get stored. This might be handled in the interim by a tied hash or an eventual phasing out of the getVars API for something that gets/sets individual VARS.

  • Similar to the above point, we need to be able to assign arbitrary metadata to various nodes. For business purposes, 18+ is one such piece of metadata. There are two types of arbitrary metadata that we'll need, "system" and "user". For instance, a category assignment will be a "system" piece of metadata (different than the current category system). A user piece of metadata might be a single vote towards, say "Best of 2012" or something like that. Multiple user pieces of metadata might translate along an algorithm to become system metadata. If enough users vote to category move something, for instance, it might get moved. That's an example of how we might use the tech, not a finalized feature.

  • Instead of deleting a node, we should have a deleted_by flag in node that means that getNode can't find it. We'd need to have a separate API for getNode to pick up on deleted items

  • Add into E2compile the ability to fall back on the previous method if the node was updated. This is predicated on the delta-encoding work toward making this worthwhile, as anything compiled is clearly more static than otherwise. Check to see whether or not on delta-encoding, we can actually update the symbol table to overwrite nodes that are compiled to disk. This would require expansion of the tool to a real library so that it can return subroutine references instead of blobs of text.

  • There needs to be a way to apply database updates in a controlled fashion, similar to Rails migrations.

Site flow changes

  • We should all of the previous revisions of a writeup.

  • Users should be able to propose edits to a writeup, as a level power.

  • Editing may be broken into a separate level scale than writing. It's an entirely different skillset; so I feel that rewarding it and entrenching it differently makes social and technological sense. We don't reward it the same way we reward writing.

Think of maybe a proposed new scale as something like this:

Level/Reqs / Writing

Level 1 / 2 WU / Voting , Homenode image

Level 2 / 5 WU /

Level 3 / 10 WU / C! ability

Level 4 / 20 WU / Unlocks Level 0 editing

Editing scale:

Level/ Reqs/ Editing

Level 0 - Ability to tag write-ups for needed revision or editing under certain categories (sources, structure, spelling, phrasing)

Level 1 - Ability to vote on change requests

Level 2 - Ability to propose written changes

Level 3 - Ability to view private feedback on nodes

Level 4 - Ability to declare something a derivative work, superseding previous writeup?

Level 5 - Ability to promote to "best of" / cull bad write-ups ignoring feedback.

The above editing levels might include a way for the administration to lock users at a particular editing level to avoid abuse, or they must approve a promotion.

Like level 2 or 3 writing scale should unlock level 1 writing. Editors will still exist, but they'll be from the pool of promotable people, and will still require a blessing from the administration. In general, I think that the levels are too steep right now, and that we need to

UI work

  • DMCA link support at the bottom of the page

  • Better header space usage. I'll need a small E2 logo for that

  • Spec out what leaderboard ads might look like for Guest User

  • Remove codehome. It's a stupid feature, seriously.

Feature work

  • We need to allow alternate writeup formatting choices, such as MediaWiki, plain text, Markdown, etc

  • There's no reason why we can't find a perl module to allow people to upload Word/RTF/whatever Docs. It's 2012.

  • We need explicit licensure types for nodes, current and past.

  • Commentary on nodes

  • Users should be able to comment on any node as a portion of the page that we'll probably call "Feedback". It will work like the current discussion section. Feedback can come in two forms, Visible to everyone and visible only to authors (and editors) .Responses should follow that viewing type (or a more restrictive one).

  • User driven usergroup nodes for collaboration purposes. Remove code-execution powers from usergroup descriptions.

  • Support writeups, instead of just filing something in Nuke Request or Bugs or something.

Something went wrong with that request. Please try again.