Guidance for using the Java API (not reinventing the wheel)

[sengsational]

I'm a long-time SikuliX IDE user, but I've never felt like the python was very tidy, so just recently I downloaded the whole artifactId oculix into Eclipse and made an Eclipse project with a dependency on artifactId oculixapi.

I asked AI to translate my Sample.py and I moved over the image files from Sample.sikulix folder and it works.

Now what I'd like to do, before I move over more of my original code is make a tidy Java framework that "does things right", such as keep artifacts segregated and organized and have ways to do things like drop in another version of an icon and have the new one, or any older one, found. I had a bunch of helper methods in python, but there was so much more that I know could be done to make scripts more maintainable and less brittle.

What I figured was someone must have written a wrapper or companion, but I didn't find one. Short of that, I searched for youtube videos, but the ones I found were not "How I configure my Java objects to use the API in a way that's easy to build and maintain."

I just started on this today, and I already have some ideas about keeping each web page as a separate class, with any images to match associated with that class. And some utility classes to run to make sure all of the resources are found before starting. I even created a Swing thread to show the images being waited on or clicked on and a countdown timer.

Anyone that's done web automation with SikuliX on web sites they don't control know that these web sites are always tweaking things, so it's a never ending effort to keep scripts running. That's why I want to see what's going on real time and be able to easily make changes to the code as changes happen to the web site.

So, any guidance would be appreciated. I suppose RTFM is in order, but way too many times I can read about something and not know how it can be leveraged to do something really helpful. That's the bit about not reinventing the wheel. If someone has some examples that leverage the API in cool ways to make it easier to make the API sing and dance, that's what I'm after.

[RaiMan]

I am @RaiMan (from SikuliX) and wish you a warm welcome in the Oculix sphere.

Some words on the history and the Python language decision
Some 15 years ago, when I took over from the MIT Sikuli, this was already decided and some good choices at that time:

• since it should work on Windows, Linux and Mac, Java was the best choice for the base (at that time)
• since it also allowed to access and automate screen, mouse and keyboard
• it should be easy to use without programming knowledge, which meant scripting (no compile and build) with instant runs
• which in turn called for the Python language with the ecosystem of Jython based on Java and direct access if needed to the Java level
• the first years the main development area was the IDE, image handling and the support for creating and running Python scripts

Later the Java API got into focus more and more, since other people wanted to integrate the features into their own dev ideas.

So my answer to your decision, to move from the Python scripting level to the Java programming level is simply: do not do it

• you have years of experience and running samples on the scripting level
• you loose the script-and-run experience (Java has a more complex eco-system and with each change needs a build-to-run)
• if you want your stuff a bit more organized: everything is available in Python language too (classes, functions, …)

… and now Oculix has a direct Python implementation: pypi oculix

So whatever you do, some basic recommendations:

• structure your work with respect to the usage targets
• have a naming convention, that reflects your structure of scripts and images
• identify reused implementations and build classes/functions

One more thing: IMHO the organization and handling of your images is vital, to minimise the overall effort. And this is a challenge already if you dev outside the SX/OX IDE, but gets even more challenging at the programming level (make, name and integrate screenshots).
So at least for the images you do best with using the IDE, which in turn means (see recommendations above) images in filesystem near your code (resources in Java).

Hope it helps.

[sengsational]

Thanks for the background and suggestions. The path of least resistance is going to be sticking with running scripts from the IDE. I've got decades of Java experience and very little Python, so I feel pretty clumsy in that environment. And although there's no compile step, there's less code validation support in the IDE (at least in 2.0.5), so finding out at runtime that I blew it on syntax can be tedious. But maybe I can get AI to help me make things more maintainable there. My scripts there have so much repeated code and I haven't been very careful managing images, so it's quite a mess. But I might be able to do a facelift.

In my one day using the Java API I really miss things like the Matching Preview and Target Offset. But I like the visual logger I made. It shows the image it's waiting for and includes a countdown timer while it waits. Then it shows the image found along side the image it was waiting for, and scrolls that down when the next wait happens. As I mentioned I'm still running 2.0.5, so maybe there are more features in the next IDE. I guess one of the reasons I got more serious about the Java path was having a better idea of how things were failing. In my Java I can watch each log entry as the process is actively running plus the image matching log. That enables me to pin point when and why a script goes off the rails.

I'm heading over to try the latest IDE. I'm taking your advice and not taking the Java road, at least for now.

[julienmerconsulting]

@sengsational — and @RaiMan's reply above is the right anchor to read first. Worth thanking him: the Python/Jython lineage he describes is the soil this whole tooling grew on, and it's still good soil today.

To genuinely take his caveats on board:

• Python + Operix (pypi oculix) is the right default for most users. Script-and-run, no build step, full IDE experience, Jython friction gone. If you haven't tried Operix on your existing sample yet, it's worth fifteen minutes of an evening before committing to the Java route.
• "More organised" is achievable in Python too — classes, modules, helpers, naming conventions, all available on the scripting side.
• Image handling is the vital topic — and that's where your second idea below lands head-on.

That said, your case has legitimate reasons for Java: existing Java team, Eclipse already set up, the wish for compile-time guarantees and IDE refactoring power Python doesn't offer in the same way. The trade-off depends on the team, and you've made the call honestly.

Quick reactions on your three ideas, language-independent:

1. One class per page, with its images as members — Page Object Pattern from the Selenium / Playwright world applied to visual automation. Right call: co-locating images with the code that uses them turns refactors from treasure hunts into mechanical work.

2. Preflight resource check before the run starts — the most precious idea in your post, and exactly the answer to @RaiMan's "image handling is vital" point. The classic pain of any visual suite is "image missing detected at minute 47". A startup pass that verifies every referenced image actually exists on disk = enormous time saved on long runs. Almost no commercial tool ships this in standard.

3. A Swing thread showing the image currently being waited on or clicked, with a countdown — a runtime test-runner UX. Useful in debug, rare in the visual-automation world.

Concrete proposal. I've been incubating a Java framework on top of OculiX along the exact same axis as your work — a Manager-based architecture (ScreenOperationsManager as orchestrator, plus ClickManager, OCRManager, RegionManager, TypeManager, WaitManager, CaptureManager), working transparently against either a local Screen or a VNCScreen for headless / remote runs. Private utilities I'd built up across long engagements, never got around to OSS-ing. Your discussion is the right nudge.

What I'd like to propose: I open a public repo inside oculix-org (provisional name oculix-java-framework, open to better), seed it with the Manager blueprint, and give you write access there. You bring the Page Object pattern, the preflight check, the live visualizer; I bring the Managers. We see where they merge cleanly. MIT, same terms as the rest of the project.

Once it's in shape, the Java framework becomes the natural Java-side companion to OculiX — the way Operix is the Python-side companion. Both lanes valid, both supported.

No pressure, no obligation — open invitation, your call on pace and scope. If after trying Operix you decide @RaiMan's Python route is the better fit, that's also a perfectly good answer.

🦎

[RaiMan]

@julienmerconsulting

Interesting - you say: I've been incubating a Java framework on top of OculiX — a Manager-based architecture. I open a public repo inside oculix-org (provisional name oculix-java-framework).
Please put me on the list of the ones who have access to the repo and might help with tests and suggestions.

One more thing for this thread:
@sengsational also talked about the valuable IDE feature to visually manage images with respect to the needed similarity score, the final borders and a related click point. This currently is only available inside the SikuliX/Oculix IDE.
IMHO this could be separated as a companion app, that in all Oculix contexts (Oculix IDE, Oculix Java, Oculix Python, …) could be used to manage images and captures visually at the side of a mature IDE (InteliJ IDEA, Eclipse, PyCharm, Visual Studio, …). Most of these IDE’s have plugin ecosystems, which then might be a target for the next years (sometimes in the past I had dreams about that :-))