-
-
Notifications
You must be signed in to change notification settings - Fork 10.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automation / Testing / Visual documentation framework #435
Comments
PS: to clarify even if we could aim to have lots of it in a separate file, that would still involve touching a lot of the core library, and some of the changes will probably lead to imgui's own public api development. |
This sound like an awesome idea. I have thought about this before as I want to document the UI API in my debugger. The API is close the exactly the same but with a different naming scheme but that is very minor. I would likely try to use this as well (given permission of course) in a slightly altered form. |
I'm bit interested in helping out with this. I wonder if you have some ideas on how to do it? That being said I don't have lots of time doing this but I want to help out at least. |
Thanks Daniel and sorry for not answering to this earlier. :) That picture is essentially showing the intent: Except it's all wrong because:
I think a first step would be to manually draft more example documentation. Which would naturally lead to creating the framework for this. It would probably embed something like the screenshot helper here https://github.com/ocornut/imgui/wiki/screenshot_tool How should the documentation be presented? When and how can we be skip context? Then we figure step by step out how we can make this work easier and more automated. Believe it or not I feel pretty bad with my English when it comes to presenting things in a consistent manner so I could do with some help :) I am not too concerned about tackling the testing part too explicitly at first. ImGui doesn't really crash so tackling this possibility in an automated manner isn't an important priority ihmo. I think for a start the documentation itself can be defacto testing. Overview style vs Single-function/symbol style I think there would be more value in providing chapters/article about series of functions rather than individual functions. There's nothing more frustrating that documentation which have a line about each functions but you never get to understand the whole picture. So I would rather have a chapters called "Menus & Popups" and say, Also, the documentation shall be seen as a general "usage guide" which include suggestions, tips, making uses of several functions to benefit from a certain pattern, etc. so I feel it is nice to not overly focus on single function/symbols. Note about repo size: I would prefer to keep the main repo small because any commit ends up in user clone. So if we end up uploading pictures they could be on the Wiki repo (which would make sense for the documentation anyway) or we create a imgui-docs repo just to host the data |
Here's a quick example I just hacked Buttons
Button() returns true when it has been pressed
Button() can takes an optional size, set individually for each axis.
A positive value sets the size
A negative value align to the edge of the window or current contents region
|
This sounds good. I also would like to separate the code examples into separate files that may or may not be included when generating the doc. That way it's possible to have allow others to use the documentation but having slightly different syntax (for example in Rust or C wrapper case) but still use the guts of the documentation. Still the generated doc can include samples with the code in question but selectable in some way. Also using markdown for the documentation would be great. |
Yes ideally it would become a set of helpers that can be used in different contexts, locally or for languages ports. It could potentially even be useful for people to document their own imgui-based helpers or even post repros here :) but we'd have to see how it evolve. Shorter term I would just suggest tackling the basic problems: parsing source code, outputting to markdown, adapting the system so that it works in more contexts. Some context may include or exclude the creation of a window, some context may need to hide some code, etc. I could create a imgui-doc repo right now and grant you access if you want to fiddle with it together and we can always scrap the repo if it ends up back inside the main repo later |
I was thinking of doing it the other way around. Parsing markdown (with inline or ref code) and generating code to be executed and then screenshoted. That way one doesn't need to parse source code as markdown is likely way easier to parse. Thoughts? |
I would guess it is easy to parse source code as long as we have known markers and don't try to support edge cases, and it's nice to write and iterate/run actual code directly in place, rather than indirectly via a MD layer. But either way would work. I semi expect the code to include various markers/oddities. (e.g. a marker to request "hiding a line" or hiding unnecessary scaffolding from from the user-visible output) so even if MD is used as source, we would have to regenerate a different MD either way. For example for a particular series of example demo we may need to setup certain conditions but maybe the setup is only visible once (or never if it's not really important in context). |
Yeah if we use markers then it should be fine. |
Created an empty repo for code https://github.com/ocornut/imgui-doc |
Thanks! I will give this some more though and come up with a suggestion. |
Sorry for the lack of updates on this one. I will try to get it done this week. |
No worry! Been myself going to a crazy patch of work from which I hope to escape soon :) Thanks for looking into that. |
Still haven't got around to do anything more about this :( If anyone else wants to help out with that would be great but my plan is still to look into it even if it will likely ta a while longer than I hope. |
Hi @ocornut , @emoon,
How badly having new macros or directives will impact the terseness & understandability of code? And, first and foremost, which ETA? Set aside the automatic testing issue (up until someone eventually solves the Halting Problem), Wouldn't be more realistic --that's a nice name alternative for IMGUI: RealisticUI btw!-- to use already existing and not-too-complex-to-learn tools ? I will state my proposal in a concrete way:
That's basically all. --R |
Doxygen is just not good enough and won't provide a testing infrastructure. Testing is the first need. There's no ETA, imgui is a hobbyist project. The aim is to make something great nothing less. If we were low-aim realists we'd be stuck with QT forever. I may have more time for ImGui from next January. It will also be an ever increasing importance to many big ImGui users (game studios) that those testing systems get implemented so perhaps they will have more incentive to help.
Those would be in the testing code, not in core imgui. Your proposal doesn't solve the testing requirements, and pardon me but doxygen is quite obtrusive. Many imgui users would be freaked out if we doxygened the code. This library is built in such a weird/specific unorthodox way that classic doxygen just isn't so appropriate. Right now I think the documentation isn't that bad for experienced programmers. As strange as it seems, it is not the current priority to make imgui accessible to less experienced programmers. Just because we don't have the bandwidth/resources to handle influx of users and solve every problems. It needs to mature slowly. And I think the library isn't finished. It may take one year, two years until it feels finished. You are right on many points, we should improve extensibility, improve examples, but that's all lots of work it doesn't happen magically. Right now I think the focus should be testing, adding useful features, improve code sharing, attract experienced programmers who can help. Accessibility will slowly improve :) I just only have so many free hours, need to make choices. Sensible contributions from imgui users are most often welcome. |
HI Omar, and thanks for the time taken for giving me a detailed answer.
I'm not trying to sell you something, so it's OK for me.
one thing I left out was: 7 . implementing a web-based playground where any developer could fiddle in code and experiments:
Agreed, but if something takes me T instead 10*T I prefer the first route nevertheless.
Really. More than one year ago I implemented a basic chart wiget to do pies and histograms, along with a calendar control. It was a day's hobbyist work as a test of what IMGUI could give me. In different github repos I see interesting components developed on modified forks. A pity that the lack of a common extension system does not allow us to summon them all. --R |
I'm putting some thoughts on this issue here after a brief discussion on twitter: So there are two major parts to this, an example code document generation framework and a regression testing framework. The idea is to make these work together so the examples are also tests, making it easier to maintain. Documentation Framework The documentation system seems doable in a fairly straight forward way using standard C++ and macros. I think the requirements would be something like:
For 1 I would use something like this example but with IMDOC_BEGIN not creating a window as I think this obfuscates the original source code. So I would have something like:
Note that this approach makes the ImGui::* code able to be pasted into code not using the doc system, and more easily searchable. Requirement 2 is simple enough using macros, as the regular imgui include can define the macros as producing nothing. To be easy to use means that the output should be a .md file and a directory with the required images in it. Preferably that file would contain all the documentation, so we may need macros to be able to include source .mds for long preamble you don't want in the .cpp file. The lightweight part is debatable, as leveraging scripting tools or lang tooling might be more powerful, however it's doable and probably faster to get the basics up and working. To be automated should mean that simply running the code gives the output with no need for user input. This would mean no default closed windows, or that the doc system can control whether a window is open or not. Regression testing Framework This is where things get tricky. I think I understand the objective - inject input and capture screenshots (we may want to capture the entire back buffer to ensure we capture changes in imgui window position) then compare against prior data and flag changes to some easily viewable output. The following code is for a demo which doesn't open new windows and where we want to test widgets in turn (range based, euto etc for brevity):
Each test function would likely need to reset state, run the demo function for a few frames, add the relevant input whilst running the demo and capture & output pngs and any required data. Code would then need to be written to compare pngs and generate a table of results. |
I'm still skeptical on this but I'm willing to give full support to your effort. |
Still working on this in the background. It is surprisingly difficult to get robust results given varying starting state. Normally most tests should run on a predictable starting state so this shouldn't be an issue, but making the high-level functions extra robust will help the system to be of use for user application, maybe even end-user exposed macros. (Still injecting mostly actual mouse/keyboard inputs.) |
Hi Omar, is this code already available somewhere? |
It isn't. I can give you access to a repo but for now it is experimental and the API will keep changing for several months. |
Yes, but only if I may start porting it on the jvm, otherwise I'd just wait |
It's way too experimental/early to be ported. |
Still working on this in the background (still very useful during development, stil too early to publish). The core imgui side of things should be visible here: https://github.com/ocornut/imgui/actions |
Hello, may I request access to the said repo with testing framework as well? I understand that its api is unstable but it would still be better than hacking internal application state for tests. |
@gavlig Please e-mail me about this. |
I found this thread trying to figure out of this project supports screen readers for the blind, which requires some kind of accessibility API. I can't figure out the answer to this but I assume this is related. Anyone know? Should go in the FAQ. If not is this planned? |
@adamierymenko On my side this is not planned at the moment. There's a little bit of overlap with the work done for this issue, but overally the bulk of the work would be to figure out and interact with the dedicated OS accessibility api which are particularly complex to deal with. Someone more experienced with accessibility need would be more a adequate fit to look into this. |
The CI integration is the cherry on top of the cake.. May I have access, Omar? I'd like to start playing around and see where it goes.. If this goes well, it will greatly help me to sync with your changes with much less pressure on my shoulders about introducing bugs/regressions, and also to find out which is the commit to blame when a test is failing (without me checking out again and again randomly commits to find it) I read here it can even run headless..! This is really nice.. |
A GIF of automation doing regression testings on some subtle tree node behaviors |
I wonder to suggest imgui that add the support of MSAA or UIA for Windows platform because it can help lots of people. |
Hey @ocornut, may I ask you also access for one collaborators of ours (@Sylvyrfysh)? |
Just a quick suggestion, which I hope is not too strange. My idea stems from markdown preview enhanced, which enables to add diagrams, equations, table of content, etc to markdown. Anyhow, I have made some test in another project, and I can obtain doc that looks like this: api.md With a doc "source" which looks like this: api.src.md A where a documented .cpp or .h file might look like this: source file with embedded markdown And the doc generation script could be inspired from this: script |
@ocornut Hey, I was wondering if I could get access the repo with the testing framework? I've recently got into ImGui over the last few months and want to try and contribute to it too |
Hello, would also be interested in testing the testing framework. |
@ocornut I would love to try out the ImGui automation testing framework, could you please add me to that repo? |
If interested please e-mail me and specify: if it is for a personal or work project, share a bit about nature of the project and how you envision automation could be useful for you. We can't let everyone in just yet but working on it. |
Hi Omar, I sent you an email few weeks ago on this matter, let me know if it's still possible to join the private repository for UI Automation Testing. Cheers |
Bump- is this still active @ocornut / @dashandslash ? |
Released Dear ImGui Test Engine + Dear ImGui Test Suite |
(EDIT 2019: Even though this is an active topic (in 2019) note that this thread started a looong time ago and the early messages are not representive of where we are going now. When the new testing framework is a mature enough to be shared I'll close this topic and open a new one.)
(EDIT 2022: Released Dear ImGui Test Engine + Dear ImGui Test Suite
https://github.com/ocornut/imgui_test_engine)
Looking for someone who would be interested in working on that, probably under my guidance.
Mentioned it in #259
The idea would be have a separate test application, could be or not on the same repo, that would A/ generate screenshots/gif + code markup output for the wiki and B/ defacto run tests.
So we could describe those documentation elements in the code, the framework would execute the test, simulate inputs to interact, take screenshots or gif, crop them as desired, parse its own .cpp file to retrieve the code (which itself could have marking in comments to hide some lines) and generate the help section that would demonstrate ways to use ImGui by showing the code and the screenshot. We would have hundreds of short examples, grouped, chaptered, tagged, indexed.
I was originally primarily interested in the value of documentation but I think the overlap with testing will be enormous and natural. Just the act of generating the documentation from runtime will be an act of testing. The testing wouldn't be unit tests at ultra fine granularity but rather functional tests (at least this is what I think they are called in the real world?): say, move mouse to corner, click and drag. We can compare screenshot, regression easily caught. Tests would be merely the unpublished part of the documentation system. Any tricky area of the code or fixed bug can be subject to a test, etc.
It's not really a trivial task to set all that up and I think it'd be highly valuable to the project. Realistically I don't think I'd be able to tackle by myself this year, so looking around if someone is interested to contribute :)
The person would be expected to be proficient in writing terse, optimal C++ and willing to adopt the general programming style of ImGui.
Thanks!
The text was updated successfully, but these errors were encountered: