Integrating Accessibility into Open Gothic

[Nuno69]

Introduction

Hello everyone. My name is Nuno and I have been working on the YAGA project which aims to make Gothic I, Gothic II/Gothic II NotR accessible and playable for blind players.
The current project is based on the Union SDK which allows me to interact with ZenGin rather easily and conveniently, however this implementation has some limitations.

1. ZenGin is a very old engine and some things are not working properly: for Yaga to function properly, I need access to the route calculation system, maps system, camera ETC. While Union gives me all the necessary APIs to work with all of that, for example it is very hard to create a reliable pathfinding system to giev a navigational aid for the player.
2. Reverse engineering: as complete as it may be, Union is still a hack on top of ZenGin, some things are still not documented, explained ETC. Of course, one can use the debugger to reverse the game's addresses but A. those tools are not accesible for me as they are very graphical and so my screenreader cannot operate them and B. I am still too inexperienced to work on such a low-level. Integrating YAGA into Open GOthic would make things much more pleasant to work with, as well as make the engine beneficial for more players, as hopefully blind accessibility would be followed by other accessibility options, such as big contrast, shade filtering ETC ETC.

Project structure

Yaga is comprised of the following distinct parts:

• ZSpeech: this is a screenreader abstraction library: blind people use different screenreading programs on their computers, whether that's JAWS, NVDA they have completely different APIs to work with. ZSpeech abstracts those APIs into a singular interface which allows me to interface with most available screenreaders.
• Readers: readers are a small pieces of code which connect to the main game loop allowing me to read things like player's health, current heading, menu, dialogs ETC.

What needs doing?

Oh boy, a lot

1. Add an ability for a blind player to select a place to go and be guided with navigational audio: a turn by turn system, where the player is guided every turn would suffice.
2. Reliable wall detection: In the current implementation, impassable obstacles (walls in short) are detected however the reliability of the feature is bad due to the engine limitations.
3. Make every interface element read: that should be easy. Even in the current state of the project many menus are read, so are the dialog options, ETC. The issue lies with the inventory system as G1 and G2 have completely different inventory systems. See the InventoryReader file in the original YAGA project to understand what I mean.

My questions

1. Would you be willing to accept any pullrequests regarding accessibility?
2. Would anyone here be willing to guide me through implementing the aforementioned features? Some things would require working sight or advanced understanding of the game development puzzle pieces which I lack for obvious reasons.
3. Do you think that implementing the pathfinding feature on the current OpenGothic's stack is reliable?
   Any input on that would be deeply appreciated. For myself, I am a huge fan of Gothic and the release of ReGoth, Union and now OpenGothic make me still believe I'll be able to play that great game myself.

[Try]

Hi, @Nuno69 !
Interesting idea, never thought that it's possible to play action games like that.

Would you be willing to accept any pullrequests regarding accessibility?

Naturally my concerns here are:

• make sure that this feature wont be something that requires regular maintenance/testing
• third-party libraries are white-listed. Noted in your code there are some *.dll usage. If possible we would like to stick to system api.
  Also if, possible some draft-PR is welcome - that will help to evaluate viability of intended changes.

Would anyone here be willing to guide me through implementing the aforementioned features?

Can do.

Do you think that implementing the pathfinding feature on the current OpenGothic's stack is reliable?
While Union gives me all the necessary APIs to work with all of that, for example it is very hard to create a reliable pathfinding system to giev a navigational aid for the player.

Union should have access to waynet, that is used by AI. Or this is not enough?

Reliable wall detection

We have ray-casting, via BulletCollision, see Camera::calcCameraColision for reference.

I'll be able to play that great game myself.

Ah, so you can validate if accessibility feature is actually working as it should.

[Nuno69]

Hi @Try.
Yes, YAGA uses various *.dll libraries because those DLLs are interfaces to popular screenreaders.
Regarding Speech we have two ways

• Stick to the DLLs: this is the most viable solution, as every blind player uses a screenreader to play, and playing with their preferred settings like voice, its pitch, rate ETC is the most optimal and that's what most games do.
• Use the System Speech APIs: For Windows, this is going to be Microsoft Speech API(SAPI 5), for MacOS it's going to be either NSSpeechSynthesizer or [AVSpeechSynthesizer (MacOS 14 Sonoma and up])](https://developer.apple.com/documentation/avfaudio/avspeechsynthesizer/). To the best of my knowledge, Linux uses something called SpeechDispatcher however I have no experience with this system whatsoever.

Regarding Waynet; yes, Union has access to it, but as far as I understand the system, objects have to be linked to the waynet as its used for something like NPC routines, script events, monster spawns ETC. This gives us access only to the objects which are considered important for the game. My idea is that the player can view the list of object that is sorted from closest to nearest and navigate to it using a turn by turn navigation system. TBT system plus a wall detection would make it possible for the blind player to fully understand the game world.
See this World Of Warcraft Classic mod or this game for the blind if you want to understand my idea.
Tell me when I can start implementing speech and what steps should I take.

[mojsior]

I agree . Implementing libraries for screen readers are very important, because alot of people are using different screen readers, sometimes with braille monitors. I think, that tolk.dll supports nvda. Usually sapi has problems with interupting speech, and we need to add a function, which after pressing a corresponding key pauses the speech. Screen readers like zdsr or jaws has this behaviour by default, but ms sapi doesnt have this as I know. I think, that you can add a support for these speech libraries, but you dont have to distribute these with the engine. We can copy these dlls if it is needed.. Wysyłane z aplikacji Outlook dla systemu iOS<https://aka.ms/o0ukef>

…

________________________________ Od: Arkadiusz Ś ***@***.***> Wysłane: Wednesday, July 5, 2023 12:25:39 PM Do: Try/OpenGothic ***@***.***> DW: mojsior ***@***.***>; Manual ***@***.***> Temat: Re: [Try/OpenGothic] Integrating Accessibility into Open Gothic (Issue #479) Hi @Try<https://github.com/Try>. Yes, YAGA uses various *.dll libraries because those DLLs are interfaces to popular screenreaders. Regarding Speech we have two ways * Stick to the DLLs: this is the most viable solution, as every blind player uses a screenreader to play, and playing with their preferred settings like voice, its pitch, rate ETC is the most optimal and that's what most games do. * Use the System Speech APIs: For Windows, this is going to be Microsoft Speech API(SAPI 5)<https://learn.microsoft.com/en-us/previous-versions/windows/desktop/ms723627(v=vs.85)>, for MacOS it's going to be either NSSpeechSynthesizer<https://developer.apple.com/documentation/appkit/nsspeechsynthesizer> or [AVSpeechSynthesizer (MacOS 14 Sonoma and up])](https://developer.apple.com/documentation/avfaudio/avspeechsynthesizer/). To the best of my knowledge, Linux uses something called SpeechDispatcher<https://freebsoft.org/speechd> however I have no experience with this system whatsoever. Regarding Waynet; yes, Union has access to it, but as far as I understand the system, objects have to be linked to the waynet as its used for something like NPC routines, script events, monster spawns ETC. This gives us access only to the objects which are considered important for the game. My idea is that the player can view the list of object that is sorted from closest to nearest and navigate to it using a turn by turn navigation system. TBT system plus a wall detection would make it possible for the blind player to fully understand the game world. See this World Of Warcraft Classic mod<https://duugu.github.io/Sku/> or this game for the blind<https://www.youtube.com/watch?v=Z4wOq-WBE7k> if you want to understand my idea. Tell me when I can start implementing speech and what steps should I take. — Reply to this email directly, view it on GitHub<#479 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHWIH2VL3D7Q4UGUNVTVHUDXOU6KHANCNFSM6AAAAAAZ56DDQE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Nuno69]

The theoretical problem with including screenreader drivers as loose DLLs is that the engine would have to connect to the internet to download them. That would require additional logi for falling back in case of the download failure. While it’s not complicated in itself, it’s the next permission that OpenGothic would have to have. Its not a problem on MacOS, since ```NSSpeechSynthesizer``` and ```AVSpeechSynthesizer``` are the system APIs, but I don’t know if connecting to the network is good.

…

On Jul 5, 2023, at 13:41, mojsior ***@***.***> wrote: I agree . Implementing libraries for screen readers are very important, because alot of people are using different screen readers, sometimes with braille monitors. I think, that tolk.dll supports nvda. Usually sapi has problems with interupting speech, and we need to add a function, which after pressing a corresponding key pauses the speech. Screen readers like zdsr or jaws has this behaviour by default, but ms sapi doesnt have this as I know. I think, that you can add a support for these speech libraries, but you dont have to distribute these with the engine. We can copy these dlls if it is needed.. Wysyłane z aplikacji Outlook dla systemu iOS<https://aka.ms/o0ukef> ________________________________ Od: Arkadiusz Ś ***@***.***> Wysłane: Wednesday, July 5, 2023 12:25:39 PM Do: Try/OpenGothic ***@***.***> DW: mojsior ***@***.***>; Manual ***@***.***> Temat: Re: [Try/OpenGothic] Integrating Accessibility into Open Gothic (Issue #479) Hi @Try<https://github.com/Try>. Yes, YAGA uses various *.dll libraries because those DLLs are interfaces to popular screenreaders. Regarding Speech we have two ways * Stick to the DLLs: this is the most viable solution, as every blind player uses a screenreader to play, and playing with their preferred settings like voice, its pitch, rate ETC is the most optimal and that's what most games do. * Use the System Speech APIs: For Windows, this is going to be Microsoft Speech API(SAPI 5)<https://learn.microsoft.com/en-us/previous-versions/windows/desktop/ms723627(v=vs.85)>, for MacOS it's going to be either NSSpeechSynthesizer<https://developer.apple.com/documentation/appkit/nsspeechsynthesizer> or [AVSpeechSynthesizer (MacOS 14 Sonoma and up])](https://developer.apple.com/documentation/avfaudio/avspeechsynthesizer/). To the best of my knowledge, Linux uses something called SpeechDispatcher<https://freebsoft.org/speechd> however I have no experience with this system whatsoever. Regarding Waynet; yes, Union has access to it, but as far as I understand the system, objects have to be linked to the waynet as its used for something like NPC routines, script events, monster spawns ETC. This gives us access only to the objects which are considered important for the game. My idea is that the player can view the list of object that is sorted from closest to nearest and navigate to it using a turn by turn navigation system. TBT system plus a wall detection would make it possible for the blind player to fully understand the game world. See this World Of Warcraft Classic mod<https://duugu.github.io/Sku/> or this game for the blind<https://www.youtube.com/watch?v=Z4wOq-WBE7k> if you want to understand my idea. Tell me when I can start implementing speech and what steps should I take. — Reply to this email directly, view it on GitHub<#479 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHWIH2VL3D7Q4UGUNVTVHUDXOU6KHANCNFSM6AAAAAAZ56DDQE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub <#479 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN6B3G2HUTGYH5A3EKO2H73XOVHH3ANCNFSM6AAAAAAZ56DDQE>. You are receiving this because you were mentioned.

[frainfreeze]

@Nuno69 not necessarily. OpenGothic could* house the code changes needed for accessibility part to work, but not concern itself with the release and testing of the accessibility part, as not to dilute the focus and dev time while more impactful tasks are still on the table. You could compile and make available "the accessible" version as standalone, independently. This would minimize the projects involvement yet keep you from having to maintain own fork.

*I'm not a contributor, so what the project "could" or "could not" do depends on its maintainers and owners.