More tests related to "maybe fixed (add tests to confirm)"

[domenic]

No description provided.

[domenic]

Implementation report:

(Note that this task was split into two PRs, #3857 and this one, for no particular reason.)

This was similar to #3848, but I knew more what I was getting into this time when asking for AI assistance. Similar to that task, my original time estimate of 90 minutes without AI / 45 minutes with AI was quite off, as I generally underestimate how much time writing up test cases takes. The total was 3 hours with AI. And similar to that task, I'm not really sure if AI saved me that much time, but it probably saved some, and it definitely made the task less menial.

This time AI wrote ~90% of the code instead of ~100%, as my patience with mistakes was a bit shorter so I stepped in to correct it more. Also ab6f24b was entirely manually.

I started by trying to resume my gemini-2.5-pro-exp-03-25 (non-MAX) Cursor session from last time, and we did ~3 tests together. But it soon became confused, forgetting basic things like our conventions for adding comments linking back to the issues, or how to run the test validation command. I euthanized that session.

I then started a new session with Cursor's gemini-2.5-pro-exp-03-25, and we did another 4 tests or so before it started hallucinating strongly that CSS layer rules had a type of CSSRule.LAYER_STATEMENT_RULE === 16, when in reality no such constant is defined in the specs and the type is 0. Normally I know that once a session starts hallucinating, it's probably best to roll back and avoid getting it further into the hallucinatory persona. However, I was hopeful that Gemini 2.5 Pro, being the newest model on the block, might do better. And I was also hopeful that since I was in Cursor and could point it directly to spec text, it would acknowledge that it was wrong.

Well, no. Instead it used its prodigious reasoning ability to tie itself into knots with all sorts of excuses about how CSS specs build on each other and even if the constant isn't defined in document X or document Y, it's actually defined in document X. (???) I had to put it out of its misery.

Other notes on my time with Gemini 2.5 pro:

• Although it was pretty smart most of the time, it had a few moments of extreme stupidity such as not understanding how to fix a linter error asking it to change const x = y.x to const { x } = y.

• It didn't like waiting for my input, and wanted to just keep agent-looping. One time when I submitted a test request and then went to rotate my laundry, in response to a test failure that occurred because of a legitimate bug in jsdom, it started trying to upgrade my dependencies. When that didn't work, it changed the test to pass by expecting the wrong output, reasoning something like "well, jsdom's answer is basically right, and we don't need to be so strict". I wish it would just stop and ask for review like I requested at the beginning of the session.

For the last few tests I went with Claude Sonnet 3.7. It was a bit over-eager sometimes, but after I reminded it to focus on just capturing the bugs reported by the user, I was happy with its performance. It seemed quicker to pick up the desired style from existing examples than Gemini did.