Extend MDanalysis AI policy for membrane-curvature (and maybe other MDAKits)

[ojeda-e]

Opening this discussion as suggested by @orbeckst in MDAnalysis/membrane-curvature#254

---

I'd like to preface this by saying that I'm aware of the MDAnalysis AI Policy and that it already applies to every repository under the MDAnalysis organization, including the project I own and currently maintain membrane-curvature.

This discussion is not intended to modify or replace that policy. Instead, I'd like to open a conversation about adding a small project-specific expectation for membrane-curvature, and possibly extend the conversation more broadly to whether MDAkits can add such policies on top of the general one in cases where a maintainer feels it would help.

Context and motivation

Over the past month, membrane-curvature has received several low-quality PRs from automated bots, along with contributions that appear to be largely LLM-generated. Two cherry-picked examples:

1. User @lovesxy-001 opened 5 PRs in less than 4 hours. The account was created the day before submitting them. See the Discord message here. The user was blocked and reported to GitHub.
2. Another PR with no prior communication that did not follow the PR template (Pull Request #243). The user seems to be submitting automated PRs across several repos.

In all honesty, I didn't think an explicit AI policy would be necessary, mainly because I'm basically the only contributor and I wasn't expecting this kind of automated volume of PRs. Lesson learned. The policy already applies to my project, but given how many of these attempts I'm seeing, it's clear it isn't being communicated visibly enough. That's something I should in my project, but before proceeding I'd like to have a discussion on possible addition to the base MDAnalysis AI Policy.

Central topic of discussion

As several devs in this organization have seen, from the very beginning of this project I've put a lot of effort into keeping the documentation at a high standard. I can confidently say that all my GSoC mentors saw how much work went into it.

Then, as the owner and maintainer of membrane-curvature, I'd like to propose a small project-level addition to the organization policy: documentation that is primarily LLM-generated will not be accepted.

Please note this is meant to sit alongside the organization policy and be slightly stricter on this one point, not to override anything.

Rationales

Membrane-curvature is a project with a scientific basis, and it's mostly used for scientific research. LLM-generated text can read perfectly well while still containing inaccuracies, and those are exactly the kind of details that are expensive to catch in review. LLMs also tend to produce odd English phrasing, which can feel out of place in a project with scientific content. In my experience, bringing AI-generated text up to the project's standard often takes longer than writing it from scratch, and review capacity is limited.

I want to clarify that I'm not against using AI tools. What I'm against is using LLM-generated text for documentation, given the low quality of the output; and low-effort contributions that are opened without even a basic review or without understanding the content, which matters even more given the scientific nature of this work.

Considering this is scientific software, I think it's fair to expect a minimum level of engagement and understanding from contributions. I don't think this is an unreasonable expectation; it defintily wasn't in the times before AI was available. What I should have done better though, is have communicated these expectations more clearly.

I'd welcome any thoughts and opinions, both from @MDAnalysis/coredevs and from other MDAkit maintainers who've had a similar experience.

Thanks.

[orbeckst]

The current version of the AI Policy defines documentation as one of the contributions that falls under the AI Policy. As such, you may not need anything extra to justify rejecting LLM-generate docs.

However, I don't see anything wrong with you being extra explicit about expectations in your project. I certainly agree with you that scientific code should have accurate documentation.

[orbeckst]

Also note this line in the AI Policy

This policy applies to all aspects of the MDAnalysis project, including all the GitHub repositories under the organization unless a project defines its own explicit AI Policy.

that lets project owners write their own AI Policy. The important point is that there exists a policy that makes expectations clear.

[IAlibay]

Two main thoughts here based on my interpretation of how we implemented the AI policy:

1. Any sub project under the organisation is welcome to implement its own AI policy to supplement/replace the main one. See the wording "unless a project defines its own explicit AI Policy". My suggestion is to start by creating a policy tailored for membrane-curvature.

2. The current AI policy as-is also does not allow for AI generated documentation, see the wording "Documentation of any kind". Alongside our exclusion of generated content, your point of not wanting low quality LLM generated documentation seems to be covered.

[ojeda-e]

Thanks for the replies. I'll add the necessary changes to communicate the expectations clearly and also refer explicitely to the MDAnalysis AI policy.
Closing this discussion and marking as resolved.

[orbeckst]

I am going to re-open this discussion so that it remains visible.

I know that its main purpose was to answer questions about one specific project but it may provide a good starting point for further discussions.