Add various selectors tests

[domenic]

These are all drawn from the issue tracker.

[domenic]

Implementation report:

I was excited and thankful that this issue was randomized to the AI-allowed bucket. It was a lot of grungy work, turning user bug reports into standardized-format test cases, and seemed perfect for AI to work on. I estimated 180 minutes without AI, and 60 minutes with AI.

However, in the end I'm not sure AI actually saved much time here. I ended up spending 240 minutes with AI. I think I might have been able to produce the same volume of tests if I were just coding with no AI, as long as I was fully locked in and coding at high speed.

I think I underestimated:

• The volume of issues I was churning through: at ~25 issues, 60 minutes is not realistic unless the AI is flawlessly doing an issue in 2 minutes without human intervention. Which might be possible if I was less picky and the AIs were slightly better at following instructions, but that was not my experience.
• The amount of iteration we'd need to get on the same page about issues like coding style, instruction following, etc.
• Just the time it takes to write the code and run it.

I was right that this task was pretty well-suited to AI, though. Unlike most other tasks so far in the study, this time I wrote ~0 lines of code. I spent the entire time just guiding the AIs through it, giving them reviews and guidance.

• I tried Claude Code first, excited because I felt this was a use case where a fully agentic setup would shine. This was going reasonably, with it autonomously browsing the GitHub issues list and writing the test, until I ran into issues where it wasn't able to run the test server from inside the Windows Subsystem for Linux environment. I wasted about 20 minutes purely trying to troubleshoot my network configuration before giving up and saying that Claude Code is not a usable product at this stage due to its lack of first-class Windows support.

• I then switched to Cursor. I went for Claude 3.7 thinking mode (not max), because I thought this might benefit from some extra iteration. This went well for like 4 or 5 tests.

  • It would do the whole flow: write the test, run it, update the expectations file if the test failed.
  • I tried at first to get it to make the commits, but it was having trouble creating multiline commit messages. (Maybe another Windows problem?) So I took over that part of the work.
  • It was sometimes quite eager, generating a lot of test cases related to the problem instead of just the simple reproduction case the user encountered. Sometimes this was nice, although I scolded it once because it failed to test the user's actual code.
  • Eventually though, it went crazy. It started hallucinating tests that didn't exist, and repeatedly trying to insert them into the expectations file, then debug the problems this caused.

• I decided to switch to trying Cursor's gemini-2.5-pro-exp-03-25 mode, the new kid on the block. This served as my workhorse for the remaining 15+ tests, and I was reasonably happy with it, but it definitely had some dumb moments:

  • I tried at the beginning to tell it not to insert redundant comments or assertion messages that duplicate the content being tested. It kept doing this and I kept having to remind it to remove them.
  • It had real troubles with lexicographically sorting the test filenames within the expectations file, which was surprisingly dumb. Like, it kept moving the line one line backward, running the tests and getting the sorting error, and repeating, instead of just inserting it into the right place.
  • I tried hard to establish a workflow of "write the test, let me suggest improvements, then run it". But it was over-eager and kept trying to run it before I had finished my checks. A bit scary as a failure of instruction-following.
  • It also sometimes added extra complications to the test instead of just testing what the user reported, but not as much as Claude.
  • Toward the end, it totally lost its marbles when I asked it to add both Selecting HTML elements became case sensitive #3750 and Nested selector should not be case-sensitive  #3544 tests to an existing test file. It hallucinated that these issues had something to do with attributes instead of tag names, and the case-insensitivity flag on attributes. I tried to correct it two or three times but it kept just iterating on this false premise.
    • I got it back on track by rejecting everything, and saying "stop. Let's start over on this problem." and then doing just one issue at a time.
    • This was strange because previously it had done two issues at a time without a problem.

I continue to be somewhat plagued by Cursor UI confusion. The two biggest issues are that committing files doesn't seem to "accept" them, and I have to do that separately or else the AI gets confused; and, typing @ then pasting links into Cursor will often have it mention the current file instead of the link text, for some reason.