Make lint fast again

[MikeBishop]

@ianswett and I were remarking that makes and commits have gotten very slow lately. Some observation of my machine shows that I'm spending a lot of that time checking for trailing whitespace and long lines.

This updates the make lint recipe to cache the hashes per branch of files that have successfully been linted and not re-lint them unless they've been modified (or you're on a new branch -- sorry). On my machine, lint goes from 12 seconds to 0.75 seconds.

[MikeBishop]

You can make it do the same for new branches too, but it requires a git post-checkout hook, which is a local configuration (unless we get the i-d-template to bring the hook along like it does the pre-commit hook). Here's the hook script:

#!/bin/bash

new_lintcache=".lintcache/`git rev-parse --abbrev-ref HEAD`";

if [ ! -d "$new_lintcache" ]; then
    for branch in `git branch --points-at $2`; do
        old_lintcache=".lintcache/$branch";
        if [ -d "$old_lintcache" ]; then
            cp -r $old_lintcache $new_lintcache;
            break;
        fi
    done
fi

[martinthomson]

I think that there is a simpler design.

Have lint depend on .lintcache/filename and generate that file as a result of linting filename. Let make do all the heavy lifting.

The downside of that approach is that make will stop after the first error. We can fix that on CI at least by using make -k on an explicit linting step.

[janaiyengar]

(sorry, nothing useful to offer at the end of a long IETF week, but I really want to rename this "Make lint fast again").

[MikeBishop]

You're right, that's a better design. Also, I can fold the branch migration into that -- check at lint time whether I can steal a lint file from another branch rather than actually running the lint commands.

I'll rearchitect that later. We should also perhaps update the Circle config to cache .lintcache as well. Does that mean I need to make the directory controllable by an env variable?

[MikeBishop]

There you go, @janaiyengar. ;-)

[martinthomson]

Caching .lintcache won't work because the code is checked out from scratch every time, so modification times are updated. Paying the price on CI isn't that bad for the moment.

[MikeBishop]

I think it should, actually, because the content of the .lint_cache file is the hash of the Markdown file. If the file hasn't changed in the latest commit, all you pay is the hashing cost to double-check that.

Make now does as you suggested.

[martinthomson]

I don't think that optimizing for re-runs in CI is worthwhile. And maybe you misunderstood my suggestion. This is much more complicated than I think we need. I'll show you what I mean with another PR.

[MikeBishop]

Closing in favor of #1032.