Your marvellously magnificent marking robot.
This is an application I use that automatically tests and marks student code assignments in Algonquin College Graphic Design’s Web Dev courses.
Built with Javascript, Node.js & Electron.
- Why a desktop app?
- Use cases
- Set up
- How the students use it
- Repo configuration with Markbot files
- Installation on student computers
- Building Markbot
- Debugging Markbot
- License & copyright
This is my second (and more successful) attempt. The first version was built on GitHub Pull Requests & Travis with MochaJS tests.
Unfortunately version one had some problems: specifically volume. In classes of 25 or more students Travis would start to choke at the end when everybody was trying to finish their work on time. There would be upwards of 25 pull requests going into Travis per minute with many requested tests to complete. It was just too slow, sometimes tests would take 20–30 minutes to complete, or just stop.
The user experience wasn’t great for the students and also a little stressful for me, so I switched to a desktop app built with Electron and (practically) the same Javascript tests.
Having the desktop app allows all the tests to be run locally and much more efficiently.
Students will fork assignment repositories from GitHub, make their changes, and drop it into Markbot. Markbot will run a battery of tests on the code and report back with the results, allowing the finalized work to be submitted & graded.
This is great for code assignments that are pass/fail—I use it in my courses. If they pass the tests then the system automatically sets their grade to complete.
It also works for non-pass/fail assignments but the grade submission component will only assign 1 point to their assignment—which I think makes sense because, I as the teacher, would then go in and do a complete assessment of their work.
There’s a few things you’ll need to do to set the repo up properly for Markbot.
- Create a build of the Markbot.app that points to your version of Markbot Server.
- Create a repo on GitHub with a
.markbot.yml
file inside—see Markbot configuration below. - I usually put
.editorconfig
and.gitignore
files in the repo to help the students not make simple mistakes. See the Markbot repo template on GitHub. - I usually make sure the repo is set up with
gh-pages
so it is a live website for the students. - If you’re using screenshot comparison be sure to use the “Develop” menu’s “Generate Reference Screenshots” for the most consistency.
- Make sure to run the “Develop” menu’s “Lock Requirements” before sending the repo to students or they will be marked as a cheater.
- Get the students to download Markbot and it’s dependencies onto their computers.
- Then students use Markbot while coding.
There is some cheat prevention build into Markbot—it’s not perfect but it’s annoying enough for students to work around that it’s just faster to do the homework.
In the “Develop” menu there’s an option named “Lock Requirements”—this should be done for every assignment or the student will get marked as a cheater.
It will hash the .markbot.yml
file and the screenshots and put the hashes into a .markbot.lock
file. If any of the HTML, CSS or JS files are marked as locked: true
they will also be hashed into the lock file.
If Markbot detects any changes to these files the user will be marked as a cheater and given a grade of 0 with Markbot Server.
Lock the requirements right at the end to make sure everything is exactly how you want it to be.
The students will fork repos then drop into Markbot which will automate the marking.
Check out this tutorial for the students to see how it works.
All the tests are set up and ready to go and inside the Markbot application. They can be configured from a .markbot.yml
file.
Place the .markbot.yml
file in the folder you want to test. Normally this is done when setting up a repository on GitHub that students would fork.
Here are the properties that you can use in the Markbot file for testing:
repo
— used as an indicator in the app screen, forliveWebsite
, & as part of the automatic grade submissioncanvasCourse
— used as part of the Canvas integration. This option will be deprecated in the future but is currently used as a trigger for submissions to Progressinator.naming
— will confirm every file & folder follows our naming conventions.namingIgnore
— an array of paths that allows certain filenames to bypass the naming conventions check.commits
— the minimum number of commits students need—will automatically subtract your commits.liveWebsite
— whether to make aHEAD
request to the GitHub URL to check that it’s accessible or not. Requires therepo
entry. If the repo isn’t set up withgh-pages
or the student hasn’t synced any commits a 404 will be issued, failing the test.restrictFileTypes
— will disallow specific extensions and file names from existing in the folder—seeapp/checks/restricted-file-types
for a list.git
— for more complex Git checks.html
— for testing HTML files.css
— for testing CSS files.js
— for testing Javascript files.screenshots
— for comparing visual differences with screenshots.functionality
— for running Javascript live tests against the website.md
— for checking Markdown files.yml
— for checking YAML files.files
— for checking images & plain text files.performance
— for checking website performance against simulated networks.allFiles
— for common requirements between all files of a specific type.
Here’s a basic Markbot file:
repo: markbot
canvasCourse: web-dev-1
naming: true
commits: 3
liveWebsite: true
restrictFileTypes: true
# Other tests, described below, would go here
Using the git
entry we can enforce some requirements on the status of the student’s Git repo.
This is a replacement for the simple commits
entry.
git:
# The minimum number of commits students need—will automatically subtract your commits.
numCommits: 2
# Force the students to make sure all their files are committed.
allCommitted: true
# Force the students to make sure all their files are pushed & synced.
allSynced: true
# Confirm specific best practices on the last 5 Git commit messages.
# Won’t prevent students from handing assignments in will only show warnings.
# (Spelling & grammar, minimum word & character length, no trailing periods, attempts to enforce present-tense imperative verbs at the start)
bestPractices: true
Use the html
entry to test HTML files, it’s an array of objects, each representing a file to test.
The path
option is the only one that’s required—leaving any of the others off will skip the test.
html:
# The HTML file’s path
- path: 'index.html'
# Whether the code in this file should be locked or not, to help prevent students from changing the code
# I use it for assignments where I give them complete HTML & they just write the CSS or JS
# With locked on, there isn’t much point providing the rest of the options
locked: true
# Whether to validate it or not
valid: true
# Check its best practices & indentation (double quoting attributes, having a <title>, indented children, etc.)
# Can be further configured in the `htmlcs.json` file
# Will be skipped if validation isn’t also checked—the document must be valid first
bestPractices: true
# Check if the headings are in the proper order and that the document starts with an <h1>
# Will be skipped if validation isn’t also checked—the document must be valid first
outline: true
# Run accessibility tests on the HTML documents using AxeCore
# It’s usually more convenient to just inherit the accessibility template which includes more tests
accessibility: true
# Can be used to test for specific elements; each entry should be a valid CSS selector
# Will be skipped if validation isn’t also checked—the document must be valid first
# If given an array, the second argument can be a custom error message
has:
- 'header nav[role="navigation"]'
- 'main'
- ['header nav li a[class][href*="index.html"]', 'The navigation should be highlighted on this page']
# You can emit warnings that won’t prevent students from submitting work using an alternative syntax
- check: 'main' # This could also be `selector: 'main'`
message: 'The `main` tag should be included for accessibility reasons'
type: 'warning'
# You can specify a limit for how many times an element should be inside the HTML code
- check: '[role="banner"]'
message: 'The “banner” role is used to define the header of the whole website so there should be only one per page'
limit: 1
# Can be used to test that specific selectors are not used in the HTML
# I would use this for ensuring that `<hr>` tags aren’t used when borders should be or that `<br>` tags aren’t used
# Will be skipped if validation isn’t also checked—the document must be valid first
# If given an array, the second argument can be a custom error message
hasNot:
- 'br'
- ['hr', 'The `hr` tag should not be used to create borders']
# Warnings too!
- selector: 'hr'
message: 'The `hr` tag should not be used to create borders'
type: 'warning'
# Regex searches on the file, for confirming specific content
# If given an array, the second argument can be a custom error message
search:
- 'Hello World!'
- ['Hello World!', 'Whoa, don’t be so grumpy, say “Hello”']
# And warnings!
- regex: 'Hello World!' # (Using `check` instead of `regex` is okay too, so everything can be consistent)
message: 'Whoa, don’t be so grumpy, say “Hello”'
type: 'warning'
# Limits work inside search to prevent content duplication
limit: 1
# Regex searches on the file, for confirming specific content isn’t found
# If given an array, the second argument can be a custom error message
searchNot:
- 'Thing-a-magic'
# Warnings too!
- check: 'Thing-a-magic'
type: 'warning'
# Confirm that the code file has less than or equal to this many lines
maxLines: 4
Use the css
entry to test CSS files, with many of the same options as the HTML.
css:
# The CSS file’s path
- path: 'css/main.css'
# Whether the code in this file should be locked or not, to help prevent students from changing the code
# I use it for assignments where I give them complete CSS & they just write the HTML or JS
# With locked on, there isn’t much point providing the rest of the options
locked: true
# Whether to validate it or not
valid: true
# Check its best practices & indentation (spaces after colons, new lines after closing blocks, etc.)
# Can be further configured in the `stylelint.json` file
# Will be skipped if validation isn’t also checked—the document must be valid first
bestPractices: true
# Can be used to test for specific selectors, properties & values
# [selector, property (optional), value (optional)]
# Will be skipped if validation isn’t also checked—the document must be valid first
has:
- ['.thing']
- ['.super', 'background-image']
- ['.thang', 'width', '50px']
# Starting the array with an `@` and a matching string will look inside media queries
- ['@38em', 'html', 'font-size', '110%']
# Adding another argument at the end will be a custom error message
- ['html', 'font-size', '100%', 'The `font-size` should always be `100%`']
- ['@38em', 'html', 'font-size', '110%', 'The `font-size` should always increase at `38em`']
# You can include all that same complexity with warnings
- check: ['@38em', 'html', 'font-size', '110%', 'The `font-size` should always increase at `38em`']
type: 'warning'
# Or a little cleaned-up version, can be used with or without warnings
- mediaQuery: '@38em'
selector: 'html'
property: 'font-size'
value: '110%'
message: 'The `font-size` should always increase at `38em`'
type: 'warning'
# Can be used to test that specific selectors do not contain certain properties
# I would use this for ensuring as little CSS duplication as possible, like forcing students to use multiple classes
# [selector, [property, property, etc.]]
# Will be skipped if validation isn’t also checked—the document must be valid first
hasNot:
- ['.btn-ghost', ['display']]
- ['.btn-subtle', ['font-size', 'text-decoration']]
# Starting the array with an `@` and a matching string will look inside media queries
- ['@110em']
# Adding another argument at the end will be a custom error message
- ['.btn-subtle', ['font-size', 'text-decoration'], 'The `.btn-subtle` shouldn’t be used']
- ['@110em', '.btn-subtle', ['font-size', 'text-decoration'], 'The `.btn-subtle` shouldn’t be used']
# Also with warnings!
- check: ['.btn-ghost', ['display']]
type: 'warning'
# Regex searches on the file
# If given an array, the second argument can be a custom error message
search:
- '@keyframes'
- ['@viewport', 'The `@viewport` should be included for the best browser compatibility']
# Warnings—woot!
- check: '@keyframes'
type: 'warning'
# Limits work inside search to prevent content duplication
limit: 1
# Regex searches on the file for confirming certain things don’t exist
# If given an array, the second argument can be a custom error message
searchNot:
- ['@media.+\(.*max-width', 'Media queries with `max-width` should not be used — use `min-width` instead']
- ['@media.+\(.*px', 'Pixel units should not be used in media queries — use `em` instead']
- ['font-size\s*:\s*.+px', 'Pixel units should not be used for `font-size` — use `rem` instead']
# Using the object syntax and `type` there can also be warning messages
- check: 'font-size\s*:\s*.+px'
message: 'Pixel units should not be used for `font-size` — use `rem` instead'
type: 'warning'
# Confirm that the code file has less than or equal to this many lines
maxLines: 4
Use the js
entry to test Javascript files.
js:
# The JS file’s path
- path: 'js/main.js'
# Whether the code in this file should be locked or not, to help prevent students from changing the code
# I use it for assignments where I give them complete JS & they just write the HTML or CSS
# With locked on, there isn’t much point providing the rest of the options
locked: true
# Whether to validate/lint it or not using a series of best practices
# Can be further configured in the `validation/eslint.json` file
valid: true
# Check its best practices & indentation (semicolons, spacing around brackets, etc.)
# Can be further configured in the `best-practices/eslint.json` file
# Will be skipped if validation isn’t also checked—the document must be valid before best practices are tested
bestPractices: true
# Regex searches on the file
search:
- 'querySelector'
- 'addEventListener'
# Using a slightly different syntax you can create warnings that don’t prevent the user from submitting
- check: 'querySelectorAll'
type: 'warning'
# Limits work inside search to prevent content duplication
limit: 1
# Regex searches on the file for confirming certain things don’t exist
# If given an array, the second argument can be a custom error message
searchNot:
- 'document.write\('
- ['console.log\(', 'The `console.log()` function should not be left in your code after you’ve finished debugging']
# Warnings work too!
- check: 'console.log\('
message: 'The `console.log()` function should not be left in your code after you’ve finished debugging'
type: 'warning'
# Confirm that the code file has less than or equal to this many lines
maxLines: 4
Markbot can be used to compare student work against reference screenshots included in the repository.
screenshots:
# The path to the HTML file that will be screenshot
- path: 'index.html'
# An array of different screen widths for taking screenshots
sizes: [400, 650, 960]
# OR: use the alternative object syntax to control the allowed percentage for each screenshot
# This will bypass the default percentages (see below)
# The syntax is `screenshot-width: allowed-difference-percentage`
sizes:
400: 10
650: 10
960: 8
# If you’d like to capture the printed version of the website, send `print` as one of the sizes
sizes: ['print']
# If there are multiple screenshots of the same path the label is required to distinguish them (it’s used in the filenames)
# It can also be used to add a little extra information to the check list
label: 'Clicked'
# By default animations & transitions are generally disabled using the * selector and !important
# This will allow you to enable them again
# Be careful to wait for them to complete before taking a screenshot
# Something like: on('.elem', 'animationend', done);
allowAnimations: true
# Execute some Javascript before capturing the screenshots
# Happens immediately, before the first screenshot is taken and is not repeated for each size
# Has access to all the same functions as the `functionality` tests with a few small exceptions:
# - `pass()` & `fail()` don’t exist
# - `done()` must be called when the screenshot capturing should begin
before: |
on('.btn', 'transitionend', () => {
done();
});
activate('.btn');
Markbot will look in the screenshots
folder for images to compare against.
The screenshots should be generated using Markbot itself for the most consistency—trigger the “Develop” menu (See Environment variables) and press “Generate Reference Screenshots”.
Markbot will display differences to students highlighted in a black & white difference image. Difference percentages are calculated and anything with a difference greater than 10%–13% is considered an error.
The difference percentage is calculated based on the area of the image, following this formula:
< 1000000px — 13%
>= 1000000px — 12%
>= 3000000px — 11%
>= 6000000px — 10%
The difference percentage is a sliding scale because as the screenshot area increased the percentage must go down to maintain a similar level of strictness.
An example of screenshot difference errors.
Students can enlarge the difference screenshot into a split view window for clearer understanding.
Markbot split view screenshot comparison tool.
Markbot has the ability to run arbitrary Javascript code against an HTML file. This is great for running integration and functionality tests on student code to make sure their Javascript is doing the right thing.
Use the functionality
entry in the .markbot.yml
file to add tests for HTML files.
Each entry in the functionality
list will perform the following actions:
- The
path
will be loaded into a hidden browser window - When the website has finished loading the testing will start
- Markbut will run through every entry in the
tests
array - Each Javascript test will be run inside a function that gets injected into the fully loaded page
- If the test calls the
pass()
function, then the test passes, otherwise it should call thefail()
function with a string describing the problem
If a single test doesn’t pass the remainder of the tests will not execute.
Here’s an example from one of my assignments:
functionality:
# The path to the HTML file to load in a hidden browser window
- path: "index.html"
# The label can be used to help distinguish tests in the check list — it’s completely optional
# When you need a full refresh of the page, including another test with the same path makes sense
# But then they’re all listed the same in the check list
# Label gives you the ability to slightly distinguish the test names
label: "Dinosaur link"
# A replacement for individual tests
# Will just confirm that there are no Javascript errors
noErrors: true
# This Javascript code will be injected into the page before it loads
# The feature exists mainly so I can overwrite built-in features like `prompt()` to test user input
setup: |
window.prompt = function () {
return 'a';
}
# An array of Javascript code pieces to run against the live website
tests:
- |
let ball = $('.ball');
let currentColour = css(ball).backgroundColor;
$('input[type="color"]').value = '#ffee66';
$('form').dispatchEvent(ev('change'));
if (currentColour == css(ball).backgroundColor) fail('The ball’s colour doesn’t change');
pass();
Each test entry will be embedded into a Javascript anonymous self-executing function with a try-catch block, like this:
(function () {
'use strict';
try {
eval("(function(){'use strict';/* Your test code will be embedded here */}())");
} catch (e) {
/* Show error messages in Markbot & console */
}
}());
Yes, eval is evil, etc. But it’s useful here to catch any syntax errors you may have in your code so they can be displayed in the debugging console.
Also notice that your test code will be wrapped in a self-executing function, this allows you to use return
to short-circuit functions when they fail
or pass
.
Your injected code will have access to a few functions to simplify what you have to write:
pass()
— Tell Markbot that this test has passedfail(reason)
— Tell Markbot that this test has failed- The
reason
should be a string that will be shown to the user in Markbot’s error list
- The
debug(...args)
— For when writing the tests, to help you debug your test code- What ever is passed into
debug()
will be written to the console
- What ever is passed into
$(selector[, target = document])
— Instead of having to writedocument.querySelector()
- The
target
parameter allows you to usequerySelector()
on elements other thandocument
—but defaults todocument
- If the
selector
isn’t found on the page the test will fail with an error message
- The
$$(selector[, target = document])
— Instead of having to writedocument.querySelectorAll()
- The
target
parameter allows you to usequerySelectorAll()
on elements other thandocument
—but defaults todocument
- If the
selector
isn’t found on the page the test will fail with an error message
- The
css(element)
— A shortcut togetComputedStyle()
bounds(element)
— A shortcut togetBoundingClientRect()
offset(element)
— Returns the complete offset for the element to thetop
andleft
of the page, usinggetBoundingClientRect()
+scrollY/X
for the calculation.- Returns an object in the format:
{ left: 0, top: 0, }
- Returns an object in the format:
on(selector/element, eventname, callback[, timeoutlength = 2000])
— This is instead of usingaddEventListener
. The problem withaddEventListener
is timing.- If the student’s code uses event delegation, but yours listens directly on the element, your listener will be fired first.
- Using
on()
will always bind to thedocument
and listen for the event to bubble back upwards—guaranteeing that your listener gets called second. selector/element
a pre-selected DOM element object or CSS selector to match for the target for your event.eventname
is any standard event likeclick
,animationend
, etc.callback
is a function that will be executed when the event is triggered. It will receive two arguments:hasError
— (bool
): whether or not the timeout was executed, meaning the event was never triggered. Set totrue
when there is an error andfalse
otherwise.ev
— the standard Javascript event object passed through.
timeoutlength
is an optional argument you can pass to control the maximum length the listener will wait to be called.
ev(eventString[, options])
— Can be used to fire an event withdispatchEvent()
- It creates a
new Event
,new MouseEvent
ornew KeyboardEvent
options
has a default of:{bubbles: true, cancelable: true}
- If you provide an options argument it will be merged with the defaults
- It creates a
hover(selector/element, callback)
— A specialized event dispatch that hovers the mouse over an element—regular JS events aren’t “trusted” and therefore won’t trigger the CSS:hover
styles.- Allows for testing to make sure student’s apply hover states to elements in CSS.
selector/element
a pre-selected DOM element object or the CSS selector of the target for your event.callback
is a function that will be executed when the hover has triggered.
activate(selector/element, callback)
— A specialized event dispatch that “activates” an element triggering the CSS:active
styles.- Allows for testing to make sure student’s apply active states to elements in CSS.
selector/element
a pre-selected DOM element object or the CSS selector of the target for your event.callback
is a function that will be executed when the hover has triggered.
send(eventname[, options[, callback]])
— sends trusted input events to the browser window. This is whathover()
andactivate()
do internally.- It’s essentially a wrapper around Electron’s
webContents.sendInputEvent()
—See the Electron docs. eventname
is one of Electron’s allowed events:mouseDown
,mouseUp
,mouseEnter
,mouseLeave
,contextMenu
,mouseWheel
,mouseMove
,keyDown
,keyUp
,char
options
allows you to set extra properties, like Electron’smodifiers
. But most importantly it sets theisTrusted
flag totrue
by default to allowhover
, etc. activate the CSS changes.
- It’s essentially a wrapper around Electron’s
Here’s an example of using ev()
and on()
:
functionality:
- path: 'index.html'
tests:
- |
let btn = $('.btn');
let btnFill = $('path:nth-child(2)', btn);
on('.btn path:nth-child(2)', 'transitionend', function (err, ev) {
if (err) fail('The transition on the button’s coloured setion never ends—check that is has a transition');
if (oldBtnFill == css(btnFill).fill) fail('The button doesn’t change colour');
pass();
});
btn.dispatchEvent(ev('click'));
Here’s an example of using hover()
:
functionality:
- path: 'index.html'
tests:
- |
let a = document.querySelector('a');
let oldBg = css(a).backgroundColor;
hover('a', function () {
if (css(a).backgroundColor == oldBg) fail('The hover colour doesn’t change on the link');
pass();
});
Markbot can check Markdown files looking for validation & best practices. It can also search inside the files for specific content.
If the Markdown files have YAML front matter that will also be validated with the same processor used by the YAML file checking.
The path
option is the only one that’s required—leaving any of the others off will skip the test.
md:
# The file’s path
- path: "README.md"
# Check validation & best practices following a specific Markdown format
valid: true
# Regex searches on the file, for confirming specific content
# If given an array, the second argument can be a custom error message
search:
- 'Dinosaurs'
- ['T\. Rex', 'Expected to see the T. Rex described']
# Warnings will work too!
# And so do limits!
# Regex searches on the file, for confirming specific content isn’t found
# If given an array, the second argument can be a custom error message
searchNot:
- 'Mammals'
# Warnings will work too!
Markbot can check YAML files looking for validation & best practices. It can also search inside the files for specific content.
The path
option is the only one that’s required—leaving any of the others off will skip the test.
yml:
# The file’s path
- path: "data.yml"
# Check validation & best practices
valid: true
# Regex searches on the file, for confirming specific content
# If given an array, the second argument can be a custom error message
search:
- 'Mammals'
- ['Dimetrodon', 'Should have explained that the Dimetrodon isn’t a dinosaur']
# Warnings will work too!
# And so do limits!
# Regex searches on the file, for confirming specific content isn’t found
# If given an array, the second argument can be a custom error message
searchNot:
- 'Dinosaurs'
# Warnings will work too!
Markbot can check random plain text files and images for specific features.
- Images (
.jpg
,.png
): compare dimensions, compare file size, check if metadata has been removed by smushing. - Text files: compare file size, check if it’s empty, search/not with regexes, check for smushing with removal of line-breaks.
- Favicons (
.ico
): — Favicons can be checked but they won’t be checked for smushing or dimensions—Markbot always enforces16
&32
pixels sizes in favicons. - SVG (
.svg
) — SVG are treated as hybrids: dimensions can be checked even though they’re technically text files (Markbot will look at thewidth
,height
, andviewBox
attributes) and smushing only checks for line breaks.
Use the files
entry to test images and text files, it’s an array of objects, each representing a file to test.
The path
option is the only one that’s required—leaving any of the others off will skip the test.
files:
# The file’s path
- path: "images/mars-2.jpg"
# Essentially checks to make sure the file doesn’t exist
# All other checks will be ignored
exists: false
# The maximum allowed file size represented in kilobytes (kB)
maxSize: 300
# For images only
# The maximum/minimum allowed width
maxWidth: 3000
minWidth: 320
# For images only
# The maximum/minimum allowed height
maxHeight: 1500
minHeight: 240
# Check if the image/file has been smushed
# For images it’ll look for extraneous metadata, something ImageOptim would remove
# For text files & SVGs it’ll just look for line breaks
smushed: true
# For text files only
# Regex searches on the file, for confirming specific content
# If given an array, the second argument can be a custom error message
search:
- '^Sitemap\:.+sitemap\.xml\s+?$'
# Warnings will work too!
# And so do limits!
# For text files only
# Regex searches on the file, for confirming specific content isn’t found
# If given an array, the second argument can be a custom error message
searchNot:
- 'Allow:'
- ['Disallow\:\s*\/', 'The disallow all directive (`Disallow: /`) should not be used']
# Warnings will work too!
# OR…
# Just pass a directory & rely on the `allFiles` directive described below
- directory: "images"
Markbot can check the performance of a website on simulated networks—or without network throttling. Markbot will check specific performance statistics and compare them to a performance budget.
The path
option is the only one that’s required. If only the path
is included the default performance budget will be used.
performance:
# The path to an HTML file to load and test
- path: 'index.html'
# The network speed (see list below)
speed: 'WIFI'
budget:
# Milliseconds for maximum load time
maxLoadTime: 1000
# Maximum number of assets
maxRequests: 15
# Maximum page size of all assets in kilobytes (kB)
maxSize: 800
# Maximum number of fonts allowed on the page
maxFonts: 5
Markbot has a few simulated network speeds built in—you can see all the details of in the app/networks.js file.
- WIFI-FAST
- WIFI-REGULAR (WIFI)
- DSL
- 4G-REGULAR (4G)
- 3G-GOOD (3G)
- 3G-REGULAR
- 2G-GOOD (2G)
- 2G-REGULAR
- GPRS
The names in the brackets are shortcuts: using speed: '4G'
is exactly the same as speed: '4G-REGULAR'
.
Here’s the default performance budget that Markbot will use if you don’t specify your own. If you leave one of the performance budget options off, Markbot will add the missing properties from the default budget.
speed: 'WIFI'
budget:
maxLoadTime: 1000
maxRequests: 15
maxSize: 800
maxFonts: 5
If you want the same values to work for all files of the same type in a project you can use the allFiles
entry. With the all files entry we can set defaults that would be applied to all the selected files.
allFiles:
# Supports any of the entries that `html` supports
html:
valid: true
bestPractices: true
outline: true
performance: true
has:
- 'h1'
# There’s also a screenshots entry to take screenshots for all HTML files specified
# Creates a `screentshots.path` entry for each HTML file
screenshots: [320, 400, 608, 960, 1440]
# You can add the `except` entry to all types to prevent all these defaults from applying to specific files
except:
- 'test.html'
# Supports any of the entries that `css` supports
css:
valid: true
# Supports any of the entries that `js` supports
js:
valid: true
# Supports any of the entries that `md` supports
md:
valid: true
# Supports any of the entries that `yml` supports
yml:
valid: true
# The functionality tests will be applied to every HTML file
# Supports any of the entries that `functionality` supports
functionality:
noErrors: true
# Supports any of the entries that `files` supports
files:
maxWidth: 2500
maxHeight: 2500
maxSize: 300
smushed: true
# Supports any of the entries that `performance` supports
performance:
speed: '3G'
html:
- path: index.html
- path: about.html
has:
- 'h2'
files:
- path: 'images/dino.jpg'
performance:
- path: 'index.html'
With this setup everything from the allFiles->html
entry would be applied to all the HTML files listed below—to help alleviate duplication.
Adding functionality
into the allFiles
entry will test that functionality on every HTML file listed in the html
entry.
With the allFiles
entry, for HTML, we can check for uniqueness between the files: e.g. unique <title>
tags or unique meta description
tags.
allFiles:
html:
unique:
- 'title'
# Using an array, the second item will be used in the error message as the description
# I use this to make the error messages a little simpler to understand
# Without the second entry, the error message will just write the selector
- ['meta[name="description"][content]', 'meta description']
- 'h1'
The above set up would force all the HTML files to have different title tags, meta descriptions and <h1>
tags.
The unique
entry expects each item to be a valid CSS selector, similar to has
& hasNot
.
If you want to check attribute content, select with the attribute selector. Markbot will grab the last selected attribute and compare its content. In the example above, Markbot is comparing the content
attribute.
Markbot has a bunch of templates inside the templates folder that your Markbot files can inherit from, thereby getting all the requirements specified in that file. Your Markbot file is more powerful and will overwrite entries—but things like has
, search
, etc. will be merged together.
The templates are just standard Markbot files, with all the same properties.
To inherit from the built-in templates add an inherit
property to your Markbot file—it’s a list of all the templates to use:
inherit:
- git-2
- html
- css
- responsive
html:
- path: index.html
In the above scenario, everything from the all those templates will be applied to your Markbot file (overwrites based on order) and therefore to the index.html
file.
You can get Markbot to ignore files within your project directory when using the allFiles
option. Include a .markbotignore
file in the same location as your .markbot.yml
and it’ll be loaded in.
It doesn’t support glob patterns only simple file and folder paths.
Here’s an example:
pattern-library.html
patterns/typography # Since this is a directory everything within would be ignored
patterns/grid
patterns/brand
patterns/icons
patterns/modules
common/grid.css
common/type.css
common/modules.css
Before getting Markbot working on student machines, these two things should be downloaded and installed on the user’s computer.
Use the Mac OS X Terminal and install the command line tools with xcode-select --install
On Windows, install Git directly from the website. When installing, on the “Adjusting your PATH environment” screen, switch to “Use Git from the Windows Command Prompt”.
Because Markbot shells out to two JAR files, the JDK must be available on the user’s computer.
Check out my lesson for the students on installing it on their computers.
Specifically starting at set 12: Student Markbot installation lesson.
Or install each tool separately, look in the docs folder for instructions.
Markbot uses Javascript, Node.js, and Electron as its build platform.
First clone Markbot to your computer using Git:
git clone git@github.com:thomasjbradley/markbot.git
If you’re using a Mac, I’d suggest installing Node.js with Homebrew.
brew install node
cd markbot
npm install
You’ll also need Wine to make the Windows version on your Mac:
brew install wine
There’s a few things you need to do to develop Markbot on your computer.
- Create two environment variables on your computer.
- Create your application config file.
- Embed the hashed version of your password into your config file.
- Download and install the dependencies.
Start by making two environment variables on your computer for the “Develop” menu and cheat locking.
MARKBOT_DEVELOP_MENU="on"
MARKBOT_LOCK_PASSCODE="some-long-password-thing"
These will allow you to enable the “Develop” menu on your computer and create locking hashes for screenshots, code files, and the .markbot.yml
file itself.
To configure your installation of Markbot you’ll need to adjust the config file.
Rename config.example.json
to just config.json
and change the following options:
ignoreCommitEmails
— (array) the list of email addresses to ignore when counting commits.
You should probably leave this setting alone unless you choose to host your own copy of Learn the Web’s Progressinator.
progressinatorApi
— (string) the URL to the Progressinator instance hosted on Learn the Web.
After you’ve created your config.json
file and created the two environment variables, run the following command: npm run hash-passcode
.
The hash-passcode
script will generate a secret
key and hash your password, embedding both into your config.json
.
The hashed passcode isn’t really for security, it only uses HMAC-SHA512. The purpose is really to be sufficiently annoying that students will do their work instead of figuring out how to cheat Markbot.
Don’t forget to copy this hashed passcode into Progressinator.
Markbot spins up a web server internally to help with performance testing and website loading. The web server uses HTTPS—though it is just a self-signed certificate. The private key and public certificate need to be generated with openssl
before Markbot can work.
In your terminal run:
npm run gen-https-cert
The script will create the appropriate files and place them into the app
directory.
While Markbot is running the tested website will be available at https://127.0.0.1:PORT
with a randomly assigned port number.
Markbot has a few external dependencies that it shells out to internally:
- Git
- Nu HTML validator (JAR)
- CSS validator (JAR)
- LanguageTool (JAR)
The vendor
folder should contain a bunch of JAR files for the HTML validator, the CSS validator and LanguageTool.
The vendor/html-validator
folder should contain the vnu.jar
—the pre-built binary works well.
Download the HTML validator release from GitHub.
The pre-build JAR files seem to be out of date, so you’ll have to compile the JAR yourself.
Download the CSS validator source from GitHub.
You’ll need the JDK and ant
to compile the validator. Install ant
with Homebrew: brew install ant
.
- Move into the directory and run
ant jar
. - Move the
css-validator.jar
file into thevendor/css-validator
folder. - Move the newly created
lib
folder and all its contents into thevendor/css-validator
folder.
The LanguageTool is used to check spelling and grammar of commit messages. It’s another Java JAR to install in the vendor
folder.
-
Go here and download the
.zip
package:(Further down the page, look for: “Stand-alone for the Desktop”)
-
Copy the following items into the
vendor/languagetool
folder:languagetool-commandline.jar
libs/
META-INF/
org/
-
Find the English dictionary, here:
org/languagetool/resource/en/hunspell/ignore.txt
-
Copy all the text from
words-to-add.txt
(it’s inside thevendor
folder) into the bottom ofignore.txt
Make sure there are no blank lines between the words.
The PDFBox application is used primarily for checking screenshots of printed versions of websites—used to convert printed PDFs into PNGs for comparison. It’s another Java JAR to install in the vendor
folder.
-
Go here and download the “Command line tools”
pdfbox-app-0.0.0.jar
package: -
Copy the following items into the
vendor/pdfbox
folder:pdfbox-app.jar
Rename it without the version number.
If you want to run a test version, you can call start
:
npm start
To build a final application for deployment to students, run build
:
npm run build
This will create both the Mac OS X version and the Windows version.
Or optionally a single platform:
npm run build-mac
npm run build-win
That’s it, Markbot should be ready to go.
Upload the Markbot installer files to a location the students can download from.
Markbot has access to the developer tools and web inspector as well as a few other tools hidden in the “Develop” menu.
To enable the “Develop” menu make sure you have the correct environment variables.
The “Develop” menu will only show when these two conditions are met:
- The
MARKBOT_DEVELOP_MENU
env var exists. - The hashed version of the
MARKBOT_LOCK_PASSCODE
is the same as the hashed version in theconfig.json
file.
© 2016–2017 Thomas J Bradley — GPL.