Creating tools

Building Tools

Throughout the IFS project, we will be integrating several tools for assessment and visualization of student work and model data. There are three streams of assessment and feedback tools, visualization, software development and writing tools.

Before you start

Currently, any tool integrated into the IFS must be called via the command line. This doesn't limit the IFS to only command line tools but indicates that any tool that requires web access must handle this connection themselves.

ALL tools should write to stdout/stderr ( for the time being) should return a JSON formatted response and report any crashes via thrown errors.

Creating Tools

The current IFS system requires that tools set for integration are command-line-tools in which output is formatted in json stdout/sterr. The easiest way to make such a tool is to write a simple Python script. Several examples exist in the tools/ directory of the project root.

Document Mark-up Tools

All tools must provide feedback in a JSON object such as the one below. Note that comments are illustrative, and are not actually allowed.

{
  "feedback": [
    // multiple feedback objects allowed below
    {
      // mandatory fields for ALL tools
      "toolName": "Display name for tool",
      "type": "Type of feedback being provided, such as 'spelling', etc.",
      "filename": "The name of the file the feedback applies to",
      "lineNum": 7,   // the line number in the document that the feedback applies to

      // optional fields for ALL tools
      "charPos": 37, // the character position of the target relative to the beginning of the document
      "linePos": 0,  // the character position of the target relative to the beginning of the line
      "feedback": "Some sort of description of why the target was highlighted",
      "severity": "Severity of the error; this is more useful for programming-type tools",
      "category": "If the error can be categorized, name the category in this field",

      // mandatory fields applicable to writing tools
      "wordNum": 42,  // the word number in the document
      "target": "word or set of words of interest; i.e. substring to be highlighted",
      "suggestions": [
        // array of suggested replacement strings for the target
      ]
      
    }
  ]
}

Empty feedback object arrays are permissible in the event that a tool does not actually provide any mark-up to a file.

Writing Tools

The feedback objects for writing tools are the most comprehensive, as target substrings must be provided to mark-up the document correctly and display relevant feedback.

Programming Tools

The feedback objects for programming tools are quite similar to that of writing tools. Programming tools should not provide a target substring however, as it is expected that the entire line will be highlighted where there is an error.

Visual Tools ( In Progress )

Visual tools can contain their own individual tags relative to the display, so long as the type-tag is specified as visual.

Statistical Tools ( In Progress )

Tools which do not mark-up the document, are permissible by providing an empty feedback array as well as a feedbackStats array of objects. These statistics are presented to the user in a table, and the associated JSON must be formatted as follows:

{
  "feedback": [],
  "feedbackStats": [
    "type": "stat",
    "toolName": "Display name for tool",
    "level": "Some indication of how easy it is to understand this statistic",
    "category": "If the statistic can be categorized, name the category",
    "filename": "The name of the file the feedback applies to",
    "statName": "The display name of the statistic",
    "statValue": 42.73 // some numeric statistic value
  ]
}

Tools may output feedback and feedbackStats arrays.