chore: add eslint-remote-tester

[bmish]

This runs all our rules on top ESLint plugin repos to ensure that they don't crash. It's like smoke testing.

https://github.com/AriPerkkio/eslint-remote-tester

[aladdin-add]

LGTM~

[MichaelDeBoey]

CC @AriPerkkio Can you review this please?

[AriPerkkio]

As repositories list is a custom one and not coming from eslint-remote-tester-repositories, I think this one could be removed. And the dependency itself.

Suggested change

const { getPathIgnorePattern } = require('eslint-remote-tester-repositories');

The default value of pathIgnorePattern is fine here.

[MichaelDeBoey]

Should we use the repositories from eslint-remote-tester-repositories instead?

[AriPerkkio]

Currently it seems that this setup runs tests against a small set of ESLint plugin repositories, as part of CI. This approach is good, and I think eslint-plugin-unicorn has similar setup. They also use a second smoke test workflow for a ~5 hour test run against randomly picked ~10k repositories.

So which ever approach is ok. Do you want to run these tests quickly for every CI run, or schedule them to run once a week for large set of repositories. Or both?

[bmish]

Thanks for explaining both approaches.

I figured we could start by running against a set of ESLint plugins, since those repos are of course more relevant and likely to trigger our rules than any other random repository. The current set is small so it's quick to run during CI.

If there was a desire to run on a very large set of repos but less frequently, I'm fine with that too.

[bmish]

@AriPerkkio I was a bit confused by this cache option. I saw some other repos disabling it. Can I just let it use the default? Or why is the default value not sufficient?

[AriPerkkio]

This flag controls whether a cloned repository should be removed from file system after it was linted. The default value true is ideal for local development, where developer may have 100s of repositories cached in file system, and they keep linting against them after modifying rule implementation frequently. Cloning repositories is slow, so caching them makes testing fast. Default values of this tool are good for local development, so some changes may be required when targeting CIs.

In this case, as there are not that many repositories listed, it might be OK to leave this as true. Github CIs have 15-20GB file systems on free tier. When linting ~10K repositories, the caching must be disabled so that we don't run out of disk space after couple of hours.