AI commit messages are low quality

[tbodt]

Commit messages should focus on explaining why a change was made, and what the outcome is, and not really worry about the details of how it was implemented. (Much has been written about this.) AI generated messages don't do this very well:

• They never seem to explain the why behind the change. (Not a surprise, this information is usually only present in the mind of the committer!)
• At the same time, they often include pointless detail about how the change was made.
• The writing style is often "fluffy" i.e. low signal to noise ratio.
• I also found one case where the message appears to have been generated off the system prompt and not the actual commit contents???? 770b0c4

To demonstrate this I scrolled through this repo's commit logs looking for seemingly AI generated messages that demonstrate this problem

e0b93f2

feat: handle null newCommit in commit creation process

Checks for a null newCommit value and throws an error if the commit
creation fails. Updates the commit outcome type to allow null values
for newCommit. Ensures the UI state updates only when a valid commit
ID is present, improving error handling and user feedback during
commit operations.

• missing any information about why this was happening and worth fixing
• yet several sentences of detail on the details of the code change
• "improving error handling and user feedback during commit operations" low SNR. feels like an attempt to explain why, except that information was just not in the context window so the model made something up.

6c03512

feat(state): add mapReduxResult to transform query result data

Introduce mapReduxResult helper to transform the data property of a
successful Redux query result using a provided function. This enables
safe and reusable data transformations while preserving the original
query result structure and status.

• again, no explanation of why
• "This enables safe and reusable data transformations while preserving the original query result structure and status." no useful information here. delete this sentence

c49525c

feat: refactor upstream status fetching and stack retrieval

Add fetchStacks method to stackService for direct stack fetching with
transformation. Refactor upstreamStatuses in upstreamIntegrationService
to be async and fetch data directly instead of using reactive queries.
Update IntegrateUpstreamModal to handle async upstream status fetching
and remove reactive derived state. These changes improve data fetching
clarity and simplify status handling logic.

• same complaint - too detailed and also not detailed enough
• "These changes improve data fetching clarity and simplify status handling logic." again, hallucinated why

---

I would conclude that the idea of fully automatic commit message generation is just not a good one. Fundamentally, a large amount of the information that should go in commit messages (why? what is the outcome? why not some other approach?) is just not in the final diff. Half the point of writing a commit message is to create a record of what the author was thinking, and that's just not possible to automate.

That said, many of the human written commit messages are also not very good! Reading through the messages in this repo made me realize there's an opportunity to have the LLM and developer work together to create real improvement over just human-written messages. Ask the LLM to check whether the commit message seems to be missing useful information, and if so generate a question for the developer to clarify the context. Then after maybe a few of these, take all the answers and generate a message based on that. (In my own uses of LLMs, I've found their real value to be as motivation, https://xkcd.com/386/ style :P)

[Caleb-T-Owens]

You are completely right here.

One area we are exploring is to try and gain more context about the code being written.

We are looking at how we could capture prompt information from agent users through a GitButler MCP server, and also as you suggest, the idea of asking the developer some questions about the changes in order to dig into the intent.