Image Recognition

[LMAzriel]

program does parse through menus but has issues finding tritium depot to start jump as well as inventory tab. ship used was type 9 and was tried with others mentioned in reame.txt. other ships had lesser results and swapping images for new crops gave worse results. test was done at reccommended 1080p at ultra graphics settings in odyssey expansion with defalut hud and keybinds. personally excited to see this program work.

fcmacros.log

[pilotso11]

I was able to reproduce your problem using Full Screen instead of Windowed mode. I took all the screen captures in windowed made as I didn't think the macro would work in full screen mode.

If you are, you can use Alt-Enter to switch between full screen and window'ed mode.

If that doesn't work, please send a full screenshot with the Right-HUD opened and "Inventory" selected so I can try and tweak the image recognition. That's where it's failing .

If it does work in windowed mode please let me know. I will see if I can tweak the images to work in both windowed and full screen mode - or have it use 2 sets of images.

[pilotso11]

I've added a UI option to control the image matching confidence, default is 75%. This is in v0.1.4. You can try adjust that. With it set at 75% I was able to get it to work on a Type-9 and in full screen mode.

[LMAzriel]

excellent thank you ill retest again and let you know. was trying to recreate your exact conditions as to how you wrote the program. it is thus far very promising.

[LMAzriel]

update to tests. it seems at 65% confidence it works to a point. got it to fuel carrier once. I originally conducted tests in windowed and it seems to be the most effective. tested on two different computers. it would appear as if field of view and low graphic settings have the best effect. also turning off gui settings as well in ED. i did try to compile an exe for ease of use but im so rusty at program tweaking its stupid lol. will continue tests. did run into an issue trying to get it to jump the carrier; the program cursors for the exit button but with help it makes it to the carrier management screen. TLDR its beginning to work on a universal level but id love to collaborate on any ideas you may have. attach a screenshot of your options>graphics page plz so i can start with a baseline and go from there. im using defaults right now.

[LMAzriel]

update to tests. all now works and screen brightness may or may not have an effect. i reccommend new images on a per person basis and that all things be absolutely default but i also reccommend low graphics so image recognition reads the pixellation more clearly. also i reccommend in graphics settings of game that the cursor be set to o.s. mouse for best effect. i only reccommend one edit here and its the use of an arrow up key to parse up to the search bar in the galmap to avoid unwanted jumps. the grid hatching of the galmap messes with the image recognition due to the opaqueness of the bar. by doing this step it gets rid of said opaqueness and immediately recognises its found the search bar. another edit i reccomend is maybe (although possibly excessive for its purpose) coding in a few restraint lines to avoid it trying to use the storage tab if at all possible. i know that the reason for the original higher image confidence was to negate this but it may allow for better flow. all in all a good progam to start.

[pilotso11]

Sorry - real life (work) has kept me busy for a few days. It's great that you got it working.

Lowering the graphics is an interesting one. My graphics settings are set to Ultra and OS Mouse as you suggested above. Didn't think to check the mouse setting, that's a good find. Also looking at it I bet the field of view setting could make a big difference in the image. (Screenshot attached).

Your comment "use an up arrow key to parse up to the search bar." I tried this. I can't get it to work reliably on my setup, and hence reverted to the mouse. But will look again at this when I get a chance - maybe make it an option.

To be more generic - as you said everyone could do their own images - though that feels quite hard. One way to make it easier might be to "Do a training session" where it asks you to point at something, then programmatically takes the screenshots as part of a setup process. That sounds cool - maybe to explore when I get another free weekend. Alternatively I could add a second - or third - set of images at different graphics settings. I was also thinking that the recognition can be tuned to only look at a part of the screen. Doing that and turning down the confidence might help.

Thanks for trying it out and all the suggestions! I guess the real test will be if someone is able to use it for part of real expedition and/or bubble-colonia travel. I might take my FC On Lost Souls 3 and if so I will definitely be using it for that! https://www.edsm.net/en/expeditions/summary/id/170/name/The+lost+souls+expedition+3

[LMAzriel]

youre ok bud. this is a hobby not a job work has alot of free time for me so i get time to do this. yeah i love the idea of the program getting a training session before use. i also had a thought of trying a black and white hud since i noticed in the coding it uses grayscale but im afraid to because im more of a troubleshooter than a programmer and i dont yet understand python. im more oldschool linux trained.

ultimately, yes im hoping to make it work for exploration. this program is answered prayer for most. i have wished i could write a program like this. it is painful jumping a carrier to colonia and back and it would be nice to have that automated. lost souls huh? ill have to look into that.

[MrWho666]

After tritium donated on carrier's tank

2022-02-25 10:50:17 - DEBUG - Looking for: images/transfer.png
2022-02-25 10:50:17 - DEBUG - images/transfer.png not found
2022-02-25 10:50:17 - DEBUG - Press: d
2022-02-25 10:50:17 - DEBUG - images/transfer.png not found
2022-02-25 10:50:17 - DEBUG - Press: d
2022-02-25 10:50:18 - DEBUG - images/transfer.png not found
2022-02-25 10:50:18 - DEBUG - Press: d
2022-02-25 10:50:18 - DEBUG - images/transfer.png not found
2022-02-25 10:50:18 - DEBUG - Press: d
2022-02-25 10:50:19 - DEBUG - images/transfer.png not found
2022-02-25 10:50:19 - DEBUG - Press: d
2022-02-25 10:50:19 - DEBUG - images/transfer.png not found
2022-02-25 10:50:19 - DEBUG - Press: d
2022-02-25 10:50:20 - DEBUG - images/transfer.png not found
2022-02-25 10:50:20 - DEBUG - Press: d
2022-02-25 10:50:20 - DEBUG - images/transfer.png not found
2022-02-25 10:50:20 - DEBUG - Press: d
2022-02-25 10:50:21 - DEBUG - images/transfer.png not found
2022-02-25 10:50:21 - DEBUG - Press: d
2022-02-25 10:50:21 - DEBUG - images/transfer.png not found
2022-02-25 10:50:21 - DEBUG - Press: d
2022-02-25 10:50:22 - DEBUG - images/transfer.png not found
2022-02-25 10:50:22 - DEBUG - Press: d
2022-02-25 10:50:22 - WARNING - Unable to find images/transfer.png after 10

[rbradh53]

I got it to get as far as looking for the "Transfer" button. It stops at the "Filter" button and selects it, then looks for the "Tritium" button. I can't get it to go past the "Filter" button to get to the "Tranfer" button.

Edit: I couldn't get the "source code" version to load the program.

[MrWho666]

I got it to get as far as looking for the "Transfer" button. It stops at the "Filter" button and selects it, then looks for the "Tritium" button. I can't get it to go past the "Filter" button to get to the "Tranfer" button.

yep. absolutely similarly

[LMAzriel]

ok so on saturday i discovered something while mapping out a search area for ahem something. you can turn off the grid on the galmap. with this discovery i decided to run further tests. i end my tests with the following recommendations.

1. game cursor must be changed to os mouse for greater accuracy and turn off all in-game gui effects. there will be some subtle effects you cant turn off so expect that you may have to continue on this list.
2. that dashboard brightness be messed with in graphics options if initial image recognition is a problem. dont mess with the fourth panel brightness!
3. in display options of galmap turn off the grid (as shown in attached picure)
4. should you have any further image recognition issues realize that the recognition works on pixel count and on color hues. NOT ON WRITING! try taking 3 screenshots at 3 second intervals to counter game ui effects then crop those images to match whats on file and ensure you name them the same name and a sequential number (ie. tritium-0 trtium-1 etc.) keep adding them in until it hits a match. mess with the size of your crop. (cropping best done with WIN-SHFT-S then pasting somewhere like discord then saving to file. probably a better way but just my personal preference). if youre superstitious and really want it to work turn your screen brightness all the way up and make sure when you do crop the image is fullscreen. dont know why but it helped me.
5. if you think this is alot of setup KEEP GOING! its worth it. this is quality gameplay FRONTIER!

@pilotso11 with these notes i congratulate you on a successful program and i close the matter. i await your next iteration and features to the program. one of which id love to see is it targeting a specific body to jump to. attached are some files that will help with image recognition a bit. expect a friend request from me and i greatfully thank you. i would love the opportunity to collaborate with you on this.
helpful images.zip

.

[pilotso11]

Thanks for sticking it out. Lesson: E:D UI setup is really flexible and that makes image recognition tough.

If it did character recognition that would certainly help, though the with the angled fonts ... Err surely someone has solved this in Python already, gears cranking!

I think the image locations are probably pretty common. Caveat on the main image which definitely varies by ship though. So a well watched setup script that steps through the UI, without the safety checks, and takes screenshots as setup might work.

Pick a body ... that's DEFINTELY going to need character recognition. Unless ... does the galmap let you search on body names? More gears cranking.

Will keep you posted.

[EuropaSteve]

I started thinking what is the same, no matter what the end users settings are?

1. The screens we need to execute commands on.
2. The commands needed to be executed.
3. Data entry required, Mouse and/or keyboard.
4. Things get brighter when selected.
5. The ratio of object placement on screen relative to each other.
   5 means that a square is a square, no matter what the end users settings are at. The distance to other squares and rectangles on the screen are held at a ratio based on the end users settings.
   To illustrate the above I’ve attached an Inkscape file of a right panel display. (Right Panel.svg)
   If layers are not displayed in the right pane of inkscape press shift+crtl+L. You can click on the eyes to show screen shots of ship panels at various resolutions and fields of view. By resizing the template rectangle over the panel, all the internal squares and rectangles line up. The ratio of object placement stays the same for all resolutions and fields of view.
   With the panel mapped and 8 tabs along the top you can tell which tab is selected by finding the average color of each tab. The selected one is the one that’s different from the others. Again this would be independent of the users settings 4) above. Press e until tab 5 is the different one. With mouse enabled screens, just click in the center of the required box.
   I found some code to mask the template boxes but not any to give X,Y,H,W. With that, I think monitoring the panel locations would be possible. This works the same for all the panels and screens that we need to use.

The Inkscape file was to large with all the Ships and resolutions I tried so I deleted a bunch.
Right panel.zip

[pilotso11]

Nice research showing that all the ships seems to be the same on the right panel.
So instead of doing the image match it could be:

• Find known location
• Use locations to select the buttons
• Use color differentiation to determine which button is selected.

That won't work for dynamic UI elements - like finding Tritium in storage, or for selecting a route, but it could work for all the static UI buttons. Perhaps with a combination of text recognition for the outliers this might minimize the image matching. Since I've got automatic image capture working to get the carrier services button off the center hud ....

[pilotso11]

Latest code build is working without any image matching. It's using a combination of known locations and text reading. Please try it out.