Guidelines for Use of AI #1920
Replies: 10 comments 26 replies
|
I agree its time for some guidelines. I think being able to explain your PR in (mostly) your own words is important, its very frustrating when someone just copies and pastes wholeheartedly the output from AI , and you know they did not bother actually checking if its a) correct and b) actually consumable by others (AI slop). I find this reverses the burden for issues and PRs, it costs nothing (in terms of time) for someone to ask AI to generate a long winded explanation for something, but shifts the burden now to those who have to read it and try a pick apart its validity (when a couple of well crafted sentences would have done the job). I have no idea how to turn this into guidelines, so this just may be me ranting..... |
|
The burden is the key point in the whole situation. It's about some balance in efforts, not just somebody being extremely lazy and others having to work equally extremely hard to try to decipher it. |
|
I've just added the human author part of 2a:
|
|
I must say that I think the above proposal is too lenient on PRs. "AI assisted" in the form of suggestion, chat discussing how to do something etc. is fine, as long as the human is the author. Whole sections created by AI is not IMO, unless we're talking about creating boilerplate starting points for the "real content". It's just too hard to judge the quality. Think of it like you're out to buy a new lawn mower (traditional, not one of those "smart" ones that spy on you). The way it has been "without AI", you can judge the quality to some extent by looking at it, you can see what mechanical solutions have been made, what materials have been used, and know quite a bit already. Combine that with what you can find out about the brand, and you're probably in a reasonable place to judge it. You don't ever have to wonder if the piston is mounted up-side-down, or if the whole engine is just a mock with a loudspeaker that makes an engine sound when you "start" it. If it was made by "AI", you can't make any of these assumptions. Most of the engine can be valid, but it might entirely lack the crankshaft. Or, it has no bearings. Or the oil channels aren't drilled all the way through. To evaluate an "AI made" lawn mower, you have to pick it completely apart, inspect every nut and bolt, check the actual design of the engine parts, the materials used, calculate the forces various parts will be subjected to, in short, you'll have to do an examination that is more laborious that it would have been to make your own lawn mower in the first place. I have no faith that reviews will be that thorough, it's not reasonable to expect. That means that we either reject all such content, or we end up letting a lot of "slop" through. I'm very skeptical that it is at all possible to quality ensure "vibe coded" content. |
|
Ultimately I don't want to make the general PR guidelines too strict. In my experience, you can generate quite good code with AI, while at the same time it's super easy to "spit out some slop". My "style" of AI coding is very supervised and usually involves only the actual code generation, architecture, interface design and these things I usually do myself. |
|
The wording sounds good to me, and it gives some guidance in a quick changing world of AI. I appreciate it! Good proposal! |
Perhaps we should also add an "AI" label (or similar) as a way of "declaring" fully AI-generated issues (and PR's). This would also make it possible for maintainers (and others) to add the label when reading something that is clearly AI-generated. I see more and more of this recently, and it would make it easier to exclude such issues and PR's when viewing lists. Personally I tend to prioritize fully AI-generated content lower, at least when I assess it as plain lazy, and when I would quickly spend more time than the person instructing the AI did. It seems we all agree so far on the requirement to disclose the use of AI (at least when fully generated, not just assisted) - perhaps we should already try to phrase this into our templates? |
|
We're now spending more time discussing templates than discussing AI-generated contributions. Not speaking on @florian-h05's behalf, i assume his point was that issues get closed when they lack the information needed to investigate the problem and the reporter also chooses not to use the template. I guess you experienced users always provide the information needed to do a proper follow-up. The current proposal is simply to extend an existing template with an AI disclosure field. Whether issue templates are ideal in every situation seems like a separate topic. Templates exist for a reason: they help less experienced contributors provide the information maintainers need and give the project a consistent process. They are not a complete solution, but they are a useful baseline. So rather than debating templates in general, I'd be interested to hear what alternative approach you would propose for handling AI-generated issues and PRs. What would work better than a disclosure field in the existing template? |
|
If you want to base it on templates, there is a point to make sure that the user actually reads what's in the template. I would at least add it as a "comment" at the very top of the template, so that the chances that the user sees it is at its best. Other than that, there's also a "preview feature" that let you create a more tailored process: https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms I haven't looked much into it, but as I understand it, you can create a sort of "form" where you can "force" a user to read/accept something before being able to actually create the issue. I think one should be careful with such measures though, nothing kills initiative like bureaucracy. But, if done with some care, it might work. To me, the most important part is to actually form the guidelines themselves. Making the users aware of them is "the next step", just having a guideline to reference if the content looks "questionable" is much better than now - where you either have to just ignore the issue, or try to explain why nobody will want to spend time on it. |
You (@mstormi) know what's annoying me? Having to ask for the openHAB version an issue is observed on. There always is a lot of UI development and having that information helps. I really wonder how long it may take someone to type in three digits and two dots ... Wrt having a label for fully AI generated issues or PRs: IMO if a issue or PR is at the point that it needs such a label, it should be closed right away. |
I am as a annoyed as you if someone doesn't provide it in an openHABian issue and it turns out to be relevant to the issue. But please do not assume that if it's not given it's because of lazyness. If not given I usually interpret it as "all" or "latest release" which I think is what most people do and mean. From a maintainer's POV that's also because if someone opens an issue, I expect him to have upgraded to latest release already and verified the issue is still there. PS . |
The problem with that is that once you look at an older issue, you can’t assume it’s the latest release, therefore giving that information is quite helpful. Alternatively, the Main UI commit (as found on the About page) can be given. I agree that snapshot numbers don’t really help, in that case Main UI commit is way more helpful. TL:DR; Just give the version number in the future and we are good. |
|
Thanks for starting this, @florian-h05. I support having clear expectations, but I’d frame them around contributor responsibility and contribution quality, not around mandatory AI declarations. AI-assisted and AI-generated code is normal nowadays, and that alone should not be the deciding factor. What matters is that the contributor is the responsible author: they understand the change, can explain design decisions, respond constructively to review, and have actually tested what they submit. For me, the bar should stay the same regardless of tooling: if a PR is low quality, untested, or the author cannot reasonably own it, maintainers may close it. If the author demonstrates ownership and quality, we review it as usual. I would therefore avoid making declaration of AI usage mandatory. Optional transparency is welcome, but enforcing disclosure is hard in practice and risks focusing on process signals instead of code quality and accountability. Disclaimer: The text above has been created with the assistance of AI, but I am fully behind the content and it 100% represents my opinion. 😉 |
|
While I really wish that OH would take a different stance on AI generated code, I get that there are too many here that's "sold" on using it, that it's probably a lost cause. I still think it would be important to know what is and isn't AI generated, I for one isn't willing to spend time on finding bugs in AI generated code, because the "normal" approach of trying to understand the author's logic and then find where it goes wrong, can't be used. There is no logic, so there is nothing to "understand". The process is focused on the result, not the way you get there, so it must be considered a "black box" that either works or don't, and isn't worth the effort trying to fix if it doesn't. But, primarily, this discussion was about AI generated issues and forum help requests. It is frankly impossible to deal with, and if the rule is going to be that "AI help requests" should be dealt with just like human ones, I think you'll find that nobody will be willing to help after some time. It is so easy, AIs will offer to file issues or formulate questions for you, and of course that's tempting for somebody - it means that they don't have to spend time trying to understand the problem themselves. But, what they produce is just so full or errors that it's usually mostly pointless. Even if as much as 50% of what's in the report is correct, how can you know which 50% that is? It's an impossible situation, and I know that I for one, am not willing to spend time on trying to decipher what are facts and what are fiction in such "reports". Up until now, it isn't really hard to spot "AI generated content" in most situations. Maybe it will be some day, but that will probably mean that the quality have gotten better so that the whole situation is less of a problem. For now, I think that the only option, if you want to allow "AI generated help requests" is to have AI answer them as well. Then they can hallucinate together and burn tokens all day long, but I doubt they'll accomplish much else. |
|
I fully understand the worry about overloading maintainers with not-thought-through code, questions and requests. I don't think there is any silver bullet on how to deal with it. Requiring "AI labelling" will only be followed by the people who probably also think through the topic themselves. People who do not care wasting maintainer's time will most likely also not care about AI-labelling their code/post/question. So eventually, it comes back to "feeling" whether something is really profound or merely slop. And yes, this is hard to tell and will become even harder in future. I just think that a mandatory labelling won't really solve the issue. |
|
It doesn't "solve" it, but it would help a lot - because when nothing is said about it, it's "approved" to post it as if it were you're own words. There are already rules about that you can't e.g. contribute work that isn't yours, there's no easy way to check/enforce this either, but it's still among the rules. It sets the standard for what is "good" and "bad" behavior. There's also a nuance here - it's not always easy to see the hallucinations/inaccuracies, and if you're a user without a lot of knowledge on the subject, it can probably be quite impossible. So, even if the user isn't "careless", he/she might just not be able to tell fact from fiction. I'm not saying that they can't post the "AI evaluation", but I think it would be very useful to know the difference between what they themselves claim/observe, and what the AI has come up with. It doesn't mean that you dismiss the AI content, it means that if you read it, you read it knowing that this probably contains a lot of inaccuracies and isn't to be read as if it were "actual information". And, you know that the information given "outside the AI block" can be trusted to be more factual. When it comes to generated code, I fear that this will be the death of many open source projects. The time it takes to come up with a huge amount of code that will take a human hours and hours to verify, is just so short. The calculus just doesn't resolve. It seems that only I alone am able to generate more code, without using AI, than what OH has resources to review 😉 I just can't imagine the mayhem we'll have when "everybody" with a credit card can easily generate huge quantities of code. The only "solution" is to let AI do the review too, and then we're practically doomed. I'm not talking about using AI to help you with syntax, catching typos or bugs, or filling out boilerplate - but letting it write the actual code. I hope I'm wrong, but I fear that only those that reject it will have a chance to survive in the long run, because everybody else will be drained of all human capital out of pure exhaustion. |
|
@kaikreuzer I've rephrased section 2 of my proposal a bit to focus more on contributor responsibility and also make it clear that low-quality PRs, PRs that don't follow our coding guidelines, etc. may be closed without further discussion, regardless of their "source". |
|
@florian-h05 Sounds good to me like this, thanks. |
|
I am also thinking about adding a short section about maintainer responsibilities:
The motivation for this is to make it clear that AI reviews don't replace human reviews, and to ensure contributors are "protected" from incorrect claims of an AI by either disclosing the use of AI or checking the correctness of the generated review. |
|
Maybe it helps to explicitly state that review comments should be in sope of the PR's goal. Others comments should explicitly be marked as optional. This to prevent raising the bar higher and higher to a level that demotivates beginners and prevents the community from growing. We should remain inclusive, every improvement should be acceptable unlesss they obviously break things. |
|
Better now? |
|
One thing I wanted to add to this conversation is that in addition to having a policy for AI, what, if anything, can be done to make it easier for contributors overall to contribute in a way that:
Not every commit and contribution is going to require super high levels of reasoning and involve subtle feats of logic. A pretty standard "interact with this REST API" type add-on should be pretty straight forward and relatively easy for a contributor to create and submit. So is there anything that can be done, in addition to setting an AI policy in documentation only, to make it easier for contributors to do it right? I'm just getting started on an experiment of having Gemini build an add-on for me, basically starting from scratch (i.e. I didn't even have Java installed). I can report back with my findings if you all think it's relevant to this discussion. Some of the ideas I'm thinking of (some of which may already be in place:
I'm literally just getting started, so my ideas are limited right now. Cline is still downloading the JDK. But I'm just thinking that if you do it for the user, there's less chance that they do it wrong compared to documenting how to do it and hoping the user follows the docs. It's just an idea I just had. May not be worth anything. |
I'm not sure this should be discussed here, but as I mentioned on the forum, I had been imagining a couple of ways to lower the threshold for simple bindings. Both require effort to implement though. One was a speicific "REST binding" that works similar to the HTTP binding, but where you can specify which endpoints exist in some simplified way and then map them to channels dynamically. The other was to have an (optional) even more "isolated" binding framework, where the author wouldn't have to worry about OH internals and only focus on the binding functionality. Making this might be more difficult than it sounds, as far as I can understand, it would have to be comlpetely single-threaded, event based, similar to how JavaScript works. But, I don't know how difficult it would be to achieve, it would require some thread and queue handling "wrapping" the binding
Although not in "Docker form", these things are mostly there already. There is a custom Eclipse installation for OH where almost everything is handled for you. Once you have followed this documentation, you have the environment you need configured: https://www.openhab.org/docs/developer/ide/eclipse.html#eclipse-ide-setup You can create a "skeleton binding" automatically to use as a starting point: https://www.openhab.org/docs/developer/#develop-a-new-binding Once that is done, it's described relatively straight forward how to handle lifecycle, state updates, commands etc: https://www.openhab.org/docs/developer/bindings/#describing-things So, many of the things you need are already in place. If you're looking specifically for a "AI way" to do this, that probably don't exist, but you can use AI to aid you anywhere in the process that is already there. Maven does the build itself for you, spotless does the linting, in short, I think most or all the things you want are there if you follow the standard installation for the OH customized version of Eclipse. |
|
Maybe to add, there is also already guidance for AIs in place, see https://github.com/openhab/openhab-addons/blob/main/AGENTS.md. |
Which is 12 pages (printed out) of manual steps to follow. The other IDEs take just as much. Almost everyone else on this thread codes Java all the time. They already have everything set up exactly the way they want it. But if you are new to it, it's a real pain in the butt. Again, this is just one data point but when I look at all that compared to asking an AI to "start a python project in a venv" in the AI extension in VSCode I have to ask myself why bother? And yes, there are lots of docs, but frankly, I know about as much about OH and how it works as anyone can know and even I found them hard to follow. If everyone thinks there's no problem, oh well. But I will say that the barrier is high. It's the reason it's taken me a decade before even attempting my first add-on. AI got me over that barrier but not in a way that's sustainable. I actually have a working add-on right now. But I can't maintain it without AI to rebuild the dev environment again. If I can't maintain it, I can't submit it anywhere fo others to use. I also found the skeleton generated to be a bit spare and not super self-explanatory. I had to go back and forth to understand what to do where and in the end just trusted the agent to do it's thing and hope it does it right.
As far as I can tell, Cline using Gemini Flash completely ignored that file. I had to feed it all that information manually, and I'm still not certain that everything documented there is correct in my add-on. At this point all I can say is:
I figured that if there were one download that had everything already configured it would make the code generated by the AIs better and lower the barriers to entry for new developers. Honestly, even after using AI, it's the only part of all of this I still don't have a good grasp on. |
|
I never meant to say that the barrier wasn't high. When I wrote my binding, I think I used a couple of days to figure out all the stuff "around" the binding before I got started on the "business logic" itself. And, I was already very familiar with Eclipse, although that wasn't only an advantage, since the "OH configured Eclipse" does quite a few things different that I usually do, so I spent some time "fighting" with that. What I tried to say, is that many of the things you ask for are there, at least partially, and that there are documentation that covers most of it (not all). To me, this is similar to many other areas of OH, lots of things are documented, but it can be hard to find the right documentation at the time where you need that piece of information. And, there's often a lack of something that "guide you through the whole process", linking to the individual in-depth documentation articles as needed. But, setting up a Python venv isn't quite comparable, because a lot of this isn't about setting up Java (that's basically done once you have installed Java), it's about setting up the build system (Maven) to fit into OH and the existing structure. OSGi and bundles for example, complicates things a lot compared to a "normal" Java project. It has benefits for OH, but it adds to the "burden" of things that make it more complicated to get started. Likewise, there's a system of XML files that define add-ons, Things, ThingTypes, Channels, translations etc. that doesn't really have to do with Java, Maven or Eclipse. It's a OH defined way of doing things, that you'd have to handle regardless of what language you use to write the binding. So, a decent chunk of the "barrier" is to get all these things in place. The "skeleton creation" is almost only about these things. What I did, was that I peeked at some other bindings, but that's a bit of "hit and miss" because while some are simple and well structured, others are not. I absolutely think that more could be done to provide a more "complete starting point", but somebody must put in the effort to make that. |
It shouldn't. Checking the documentation at https://docs.cline.bot/customization/cline-rules#supported-rule-types, it specifically names AGENTS.md as a supported input. |
Maybe the introduction of DevContainers could help a lot with setting up a dev environment. It would allow us maintainers to specify which tools in which version are needed for development. IDEs supporting DevContainers (many, incl. VS Code) usually recognize the manifest automatically and offer to get the dev environment up and running automatically. |
Indeed. But as it did its thing and started making decisions, setting stuff up and writing the code it was clear that it either had not read it or forgotten what was in it. I had to tell it again. 🤷
It certainly would have helped me a lot I think. It still would. |
|
FWIW, I think the Linux Kernel team has defined some decent guidelines for themselves: https://docs.kernel.org/process/coding-assistants.html |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi @openhab/maintainers,
as we have already received a number of bug reports across our GitHub repositories that contained incorrect AI claims, discussion emerged on the community (https://community.openhab.org/t/ai-generated-bug-reports-github-issues-etc) about the introduction of openHAB-wide guidelines on AI-generated contents like bug reports (GitHub issues) and PRs.
Based on @rkoshak's proposal in https://community.openhab.org/t/ai-generated-bug-reports-github-issues-etc/169417/3, I have the following proposal:
Please share your thoughts on my proposal.
Also inviting @Nadahar to the discussion.
Best,
Florian
All reactions