genshingrab

A python script to specifically scrape text data shown on the HUD of Genshin Impact.

---

genshingrab is inspired from Kxnrl's GenshinDiscordRpc github issue where the issue starter and the dev said, quoting them:

ScarVite:
You could implement this using the small image and for hover text the level
Every Char could have a Different Image, which represents the character, just as FF XIV is doing it

Kxnrl:
No, we can not access game process memory.
This is an external tool.

Kxnrl further says it is possible, but it also increases the possibility the game anticheat (presumably the mhyprot2.sys) will trigger. Henceforth, I think would be good if I try to give them a temporary solution for him.

If you wanted to try:

1. Clone this repository to your local system which you used to run Genshin Impact
2. Install the required dependencies (listed in Dependencies section)
3. Run Genshin Impact in full screen mode
4. Go back to desktop and run this using elevated (run as administrator) Command Prompt in the project folder everytime you want to get your roster/party setup names:
   • If using only required dependencies:

     python grab.py

   • Or if you also install all the optional dependencies:

     ipython --no-banner grab.py

5. Quickly switch into Genshin Impact again within 1 second after the code started
6. Keep playing
7. When it's done, you will be automatically alt+tabbed to see the last screenshot. The roster names will be shown on Command Prompt that you used to run the code.

Current limitations

For now, genshingrab:

• only supports to read the roster/party setup.
• current iteration is takes assumption that Genshin Impact is running in full screen mode
• only works for Windows version of Genshin Impact, and there's no plan to support other platforms.
• program must be run manually every time you want to get roster/party setup, and you have to quickly alt+tab to the game within 1 second after running the program
• probably will not run smoothly in low-end computers while playing Genshin Impact due to the complexity of the algorithm

Some of the explanations regarding these limitations will be put in the Disclaimer section.

How it works

For now, genshingrab In addition, only works for Windows version of Genshin Impact.

genshingrab (in the current design plan) works by screenshotting the game every 5 minutes (though in the current commit, it only run once.), in which every time it triggers (which will be refered as 'the periodic refresh'), will screenshot the game for 20-100 times in really short interval possible between screenshots.

Those screenshotted images on a periodic refresh will be passed into an Optical Character Recognition (OCR) algorithm to be scanned for texts. Each scanned texts from each screenshots will be merged into one long array, and then counted. The top four of the names with highest frequencies are the valid roster names. The pseudocode looked like this:

every 5 minutes:

    screenshot_list = []
    ocr_data_list = []
    name_list = []

    execute this from 20 up to 100 times:

        take screenshot from game then put it to screenshot_list
        generate ocr_data from every screenshot in screenshot_list then put it to ocr_data_list
        filter name from every ocr_data in ocr_data_list then put it to name_list

    unique = []
    counts = []

    put all the unique name from name_list to unique
    count the occurences of the unique name in the name list then put it to counts

    sort unique based of their counts, from highest frequency to lowest frequency

    return the first four names in unique

Dependencies

Currently, this works for python 3.8

I advise you to first use an virtual environment for this python project, as some of this may conflict with your current python libraries. You may use virtualenv or Anaconda's conda to achieve that.

Python dependencies

These dependencies are strictly required. Listed in requirements.txt:

• opencv-python == 4.4.0 (Used for image processing, specifically to preprocess images before passing through Tesseract)
• numpy == 1.19.2 (Ditto (same as above))
• Pillow == 8.0.0 (Ditto (same as above))
• pywin32 == 228 (Capturing the window output in Windows. Probably works for Windows 7 and 8, but surely works on 10)
• pytesseract == 3.6.8 (To bridge the communication between python and Tesseract)

To install them, execute this on the project root folder:

pip install -r requirements.txt

After it is done, you may still have to go to your python or python virtual environment folder to run a script in order to complete the installation for pywin32 dependency. In said folder, run:

python Scripts/pywin32_postinstall.py -install

Windows Dependencies

This is

• Tesseract == 4.1 (To scan the images for texts)

In order to complete the installation of Tesseract, make sure it is run-able from the Command Prompt (visible from the Windows PATH environment variable):

tesseract

If it is not detected, add your installation folder to the Windows PATH variable.

Note that by default, Tesseract will install in english. You can install for other languages that uses different glyph

Optional Dependencies

These dependencies are not strictly required, and mostly only to ease your development.

• jupyter == 1.0.0 (You can interactively code using iPython shell or just straight experimenting using Jupyter Notebook)

Disclaimer

As from what mentioned earlier, this project only support to read your roster/party setup for now. Though the target of this project is to make sure third party developers could easily access the game content while MiHoYo still does not implement developer-friendly SDKs (which will be very unlikely), this project has to start somewhere and I choose the use case in the isue mentioned in the beginning, which is the party setup and the current location, and starts with only the party setup first.

Also during the first implementation, I am working only on my computer first to focus to at least make a proof of concept, which works. The code, I'll be honest is a spaghetti, if not a jungle. Therefore, please expect some bugs or problems, and when possible we're going to try to broaden the use case. Also, I may have not accounted the worst scenario, i.e using low-end computer to run this, because Image processing is famous for its computational costs.

I don't have a full grasp of pywin32 library, and that causes to force the players to open Genshin Impact 1 second within the program is running. It is also possible that the program will not run as expected if you alt+tabbed while it is processing. This code will definitely work for windowed mode though, but the user experience will not be nice either because this code forces to make sure Genshin Impact window is on the top of all other windows.

In no way genshingrab is affiliated to MiHoYo, Genshin Impact, Google, and OpenCV, references regarding any of them are under their respective rights.

Informations obtained by this program regarding the data scraped from the screenshots of the Genshin Impact client are never meant to change the gameplay which is unfair for other players or harmful for the affiliated company as this data never processed for the usage to divert the inner workings of Genshin Impact. Genshin Impact experience will stay their way. This program will not inspect, change, add, or decrease the inner workings of Genshin Impact.

The true intention of this program is made to make sure third-party developers that are building projects related to Genshin Impact may obtain the data from the consenting user for a fair and safe usages and not violating terms of serives of Genshin Impact, including but not limited to these use cases:

• Source of Discord Rich Presence data which should be public,
• Tracking progress of the users i.e day-to-day basis level progress,
• Display player details on their livestreams i.e on YouTube or Twitch, and
• Creating heat map of in-game locations of a user so they know which place they often or rarely visit on last session,
• etc. you name it,

without altering Genshin Impact itself in many ways.

The program in this project is not collecting any data in a permanent storage which is accessible by the developers, other users, or any third parties except if the user consented to share the data explicitly. This is simply a tool to help third party developers to create more helpful tools either for developers or end users.

We are also hoping that MiHoYo would open their access for public API or at least manage integrations to other platforms in which developers can build on it without fear of being

Acknowledgement

Thanks for Kxnrl for the amazing project which I use almost everytime I play Genshin Impact. I hope that project will finally be completed for all use cases and the experience in using the Discord RPC be seamless.

Also thanks for my friends both on Genshin Impact and Facebook friends list for the support.

And Hana Macchia for the entertainment you gave to us all, or at least just me, during the development.

Michael Reeves. Why. Jokes aside the fortnite video is basically helping me creating this lol

MiHoYo, for the amazing game!

Developer Contact

me lol me@maufirf.me or Discord maufirf#8262 (I don't have server yet lol, i'll consider it)

How unprofessional I am, right?