Indic Scripts Including Hindi, Telugu, Tamil and many more not rendering as they supposed to be. (464) #431

GalacticEmperor1 · 2023-02-12T08:35:47Z

**Issue №464 opened by shubhampateliitm at 2018-06-29 10:31:37**

make unit test for pygame to check if rendering works.
SDL_TTF is released with patch (2.0.16 should have patch included).
expose TTF_SetDirection TTF_SetScript functions.
windows pygame build that includes harfbuzz.
manylinux pygame build that includes harfbuzz.
mac wheel build that includes harfbuzz.

When I am trying to render Hindi text using render_to(). It is rendering inappropriately. for eg.
which should be written like
. Also like
which should be displayed like
.
Also there is a stackoverflow question that raise the similar issue. https://stackoverflow.com/questions/44254171/devanagari-text-rendering-improperly-in-pygame
Moreover there is a bugzilla post that discuss similar issue..
Can someone fix this bug ??

Related Docs: https://www.pygame.org/docs/ref/font.html

Comments

# # rohitsaluja22 commented at 2018-07-09 05:39:38

Hi, Every font file comes with a gsub file which is a set of rules that define substituting consegutive glyphs with a joint glyph for example. Pygame ignores such gsub files while rendering font. Can anyone tell how to force pygame to use gsub rules?

When gsub rules are not used, in Devanagari "म्ह" looks like "म् ह"(ignore space) as shown in the image:-

# # illume commented at 2018-07-14 13:34:43

Ah ok. So it seems the patch on the libsdl bugzilla for SDL_ttf (posted in 2015) is for using HarfBuzz. I haven't looked at the patch, but it seems getting someone to work on it in libsdl would be one way to get support there.

# # bedapudi6788 commented at 2019-01-12 08:06:03

Has anyone been able to get this working?

# # peanutbutterandcrackers commented at 2019-08-23 01:22:12

I can confirm that the issue still exists in pygame-1.9.6 as of today (and that PR # 706 does not actually fix this issue).

Code:

import pygame, sys
from pygame.locals import *

pygame.init()
DISPLAYSURF = pygame.display.set_mode((400, 300))
pygame.display.set_caption('Hello World!')

WHITE = (255, 255, 255)
GREEN = (0, 255, 0)
BLUE = (0, 0, 128)

text = "प्रफुल्ल"

fontObj = pygame.font.SysFont('Lohit Devanagari', 32)
textSurfaceObj = fontObj.render(text, True, GREEN, BLUE)
textRectObj = textSurfaceObj.get_rect()
textRectObj.center = (200, 150)

while True:
    DISPLAYSURF.fill(WHITE)
    DISPLAYSURF.blit(textSurfaceObj, textRectObj)
    for event in pygame.event.get():
        if event.type == QUIT:
            pygame.quit()
            sys.exit()
    pygame.display.update()

Testing Notes: 'Lohit Devanagari' font can be installed on any ubuntu-based distro by installing the package fonts-lohit-deva.

@dlon, @husano896: Esteemed developers, please pardon my pinging you, but according to the stackoverflow discussion:

I think is a SDL_ttf problem (the underlying component which actually renders the text).

Could you please look into it and make sure whether or not that is the case and let me know so that I can perhaps raise an issue over there? Please forgive me I do not have the knowledge to debug this further.

If fixed, it would be a massive help to us around here in the Indian-subcontinent. And we would be very very grateful. I hope to get your guidance in this.

Thank you for your time.

# # peanutbutterandcrackers commented at 2019-08-23 01:27:46

@ilario-pierbattista - Please pardon my abrupt pinging you, good sir, but because I saw this gist by you, I thought that, perhaps, you might also have something that you could bring to the discussion here. I am sorry I do not have the knowledge to understand the inner details of SDL, or to even understand the gist.

# # peanutbutterandcrackers commented at 2019-08-28 07:02:55

@illume @takluyver @sublimator @bitcraft @dlon - Please excuse my impatience, esteemed developers, but since this was not getting any attention, I am taking the liberty of pinging the pygame project members as listed on GitHub. I hope that perhaps some of you might be able to shed some light into this. Thank you

# # illume commented at 2019-08-28 12:54:17

@peanutbutterandcrackers Please respect that people have other things going on, and this is a volunteer project. If no one is able to answer right away it's usually because they don't have time at the moment.

# # peanutbutterandcrackers commented at 2019-08-29 04:01:04

@illume - Yes, sir. I understand. I am sorry. I just wanted to get some information regarding whether I should be actually reporting this issue to the SDL devs instead of pygame devs, and if I could get the confirmation, I could go over there.

# # illume commented at 2019-08-29 07:32:18

Hi,

No worries. I understand the feature is important for you(And others!).

The issue is reported, but really the next step is figuring out the best
way to get it done.

As mentioned in an earlier thread, there is an open bug with an old patch
for SDL_ttf on the SDL bug tracker. Maybe this is the best way forward... I
don’t know.

For pygame we need to get a windows build, and various other builds of
“harfuzz”, and then configure them for every platform. But maybe this isn’t
the best library to use in 2019. We’d need feedback from developers
familiar with the font rendering landscape. Like, is this library available
and packaged on all platforms?

But who can do this work, or how can we get this patch into SDL_ttf?
Probably a good step is seeing if that patch still applies, and then seeing
if it compiles and works. This would reduce the work of the SDL people.
Then asking for feedback from them on what further work the patch needs.

There’s lots of small steps in getting this into a build and working
correctly left to do. We need a volunteer for this however. I can help
guide someone in how to get this done, but have too many things I need to
do at the moment to take on the work myself.

One small step that only requires python knowledge would be to write a test
case, as a pygame unit test. This way it will be easier for pygame
developers (and SDL devs) to see that it is working. Would you be
interested in helping out with this? I could guide you on the way.

cheers,

On Thursday, August 29, 2019, peanutbutterandcrackers <
notifications@github.com> wrote:

@illume https://github.com/illume - Yes, sir. I understand. I am sorry.
I just wanted to get some information regarding whether I should be
actually reporting this issue to the SDL devs instead of pygame devs, and
if I could get the confirmation, I could go over there.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/pygame/pygame/issues/464?email_source=notifications&email_token=AAACKRKDGNM2SBO3B2DGTV3QG5CYBA5CNFSM4FHTG572YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5NEY7Y# issuecomment-526011519>,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAACKRKUZVUERYH7C3U5MU3QG5CYBANCNFSM4FHTG57Q
.

# # peanutbutterandcrackers commented at 2019-08-29 07:56:00

@illume - Yes, sir! I would be honoured! Any little thing I could do to help, I would like to do it. However, I must admit that my knowledge regarding the matter is next-to-zero and I am not the very quickest of learners. Yet, I will try my best: even if it takes a lot of time. :)

# # illume commented at 2019-08-29 09:10:27

I left a note about this issue in the stackoverflow post about this issue.

# # illume commented at 2019-08-29 09:13:59

@peanutbutterandcrackers cool! Thanks. For a test we need an assertion to pass or fail if the code is correct.

Can you modify the code you posted so that it would pass/fail if the rendering is correct?

Something like this where you have to modify the is_correct code for your test:

is_correct == surf.get_at((x, y)) == (255,255,255,0)
print("is correct:%s" % is_correct)

# # illume commented at 2019-08-29 09:15:35

I just checked the SDL issue, and it has been merged into SDL_ttf (but disabled by default).
See https://bugzilla.libsdl.org/show_bug.cgi?id=3046

However it hasn't made it into a release of SDL_ttf yet. The last release(release-2.0.15) is 7 months ago and the patch was merged 4 months ago.

I linked this issue to the libsdl one.

# # illume commented at 2019-08-29 09:26:32

Here are the new C functions exposed by SDL_ttf (from this link).

/* Set Direction and Script to be used for text shaping.
  - direction is of type hb_direction_t
  - script is of type hb_script_t
This functions returns always 0, or -1 if SDL_ttf is not compiled with HarfBuzz
*/
extern DECLSPEC int SDLCALL TTF_SetDirection(int direction); /* hb_direction_t */
extern DECLSPEC int SDLCALL TTF_SetScript(int script); /* hb_script_t */

# # illume commented at 2019-08-29 09:35:04

Mac harfbuzz exists in homebrew.
Debian linux https://packages.debian.org/search?keywords=harfbuzz is available from oldstable (jessie).
Windows, the harfbuzz website says it has windows binaries. https://www.freedesktop.org/wiki/Software/HarfBuzz/

# # illume commented at 2019-08-29 09:39:03

I made a todo list at the top of the issue with some of the things that need to be done.

# # peanutbutterandcrackers commented at 2019-08-31 04:01:05

@illume - Please pardon my delayed reply, good sir. Great! I am working (right now) on something that I hope will be even more versatile (the test-script, I mean). I will probably upload it to GitHub sometime today and will post the link here. Thank you for your patience.

# # peanutbutterandcrackers commented at 2019-08-31 17:20:11

@illume - I have just pushed the project to GitHub. You can find it here. It is horribly ugly. But I will be re-iterating through it in the coming days. I hope the README.md file is clear-enough as to it's usage.

# # peanutbutterandcrackers commented at 2019-09-26 06:23:51

An Update: The test script now works with GIMP 2.8.x as well as GIMP 2.10.x. I have been updating/polishing it up sporadically. Please do check it out and let me know if there is anything I should update. It would be nice, too, if some of the reporters could add other Indic-scripts in the text.json file (Telugu, Tamil, etc.); for now it only tests Latin and Devanagari. Issues and PRs are welcome. If there is anything that is not understood, please do let me know so that I can update the README.md with further documentation/explanation.

Now we only need to wait for SDL_TTF 2.0.16 before this issue-resolution catches momentum.

P. S: Code-reviews are most welcome!

# # illume commented at 2019-10-07 19:01:48

Thanks @peanutbutterandcrackers

I had a quick look at it, but haven't had a chance to play around with it yet.
But it does seem like it is in a good shape to help whoever wants to work on this.

cheers!

# # peanutbutterandcrackers commented at 2019-10-09 08:04:37

@illume - Great!

Regarding "whoever wants to work on this": if there isn't one (and we're counting on sdl_ttf 2.16 to be released as the next step in this flow, anyways), could you recommend me some books that I could study to be able to work on this? As in, all the books/documentations/videos/lectures/etc. that an intermediate python programmer (who is unfamiliar with C) should go through in order to become competent enough to contribute in this task. And pygame in general. I would love to have a road-map that I could follow. In order words, how does one become as competent as you?

# # illume commented at 2019-10-19 14:28:33

Hi @peanutbutterandcrackers,

sorry I didn't respond earlier.

I couldn't think of a good answer to your question. But here is a list of good learning resources...

https://www.reddit.com/r/C_Programming/ (good forum of C developers)
https://www.pygame.org/wiki/Contribute (here to get started on pygame things)
https://www.pygame.org/wiki/Hacking (some tips for working with pygame)
https://renesd.blogspot.com/2019/09/post-modern-c-tooling.html (some newer C based tools)
Modern C book https://gforge.inria.fr/frs/?group_id=6881 This book apparently is good if you haven't encountered C before. I'm not sure how successful it is at it, but I hear from a few others that it's pretty good. It's also available for a free download.

Depending on what OS you are on, I could try and find other links.
Feel free to ask questions, and I'll try and answer.

cheers!

# # peanutbutterandcrackers commented at 2019-10-20 04:21:53

@illume - Thank you! I am on Linux Mint 19, currently. I will be looking at the resources that you have provided. Thank you very much!

# # rohitsaluja22 commented at 2019-11-02 19:23:38

For Devanagari I have posted the solution in last part of Readme here (follow 7.0 and 7.1):

https://github.com/rohitsaluja22/OCR-On-the-go

# # illume commented at 2021-05-30 02:00:18

SDL has moved issue trackers: and the new related bug is here: libsdl-org/SDL_ttf#62

# # Starbuck5 commented at 2022-03-20 09:14:00

I think I've managed to get something working on this, thanks to SDL_ttf 2.0.18

Rendering the string "प्रफुल्ल"

pygame 2.1.3.dev4

pygame with SDL_ttf 2.0.18 and TTF_SetScript hardcoded to devanagari

Not quite sure how to expose the functionality though. It needs to boil down to these eventually: https://harfbuzz.github.io/harfbuzz-hb-common.html# hb-script-t. They all have a 4 character code that can be put into a 32 bit int. It would be a lot of constants to add to pygame for every supported script, we could have people input the 4 byte code manually.

font = pygame.font.Font("Something that supports hindi characters.ttf", 20)
font.set_script("Deva")

# # prajwalgatti commented at 2022-07-20 12:51:43

@Starbuck5 Can you explain a bit more on how you achieved this?
"set_script" is not a method of "pygame.font.Font" is the error I get when I tried your solution.

Edit: I think your comment was more of a proposal, I realise that now. How do I hardcode TTF_SetScript to Devanagari? Appreciate any help you can offer. Thanks!

# # Starbuck5 commented at 2022-07-22 06:51:58

Edit: I think your comment was more of a proposal, I realise that now. How do I hardcode TTF_SetScript to Devanagari? Appreciate any help you can offer. Thanks!

Yeah, @prajwalgatti, I did a custom build of pygame with code changes to get this to work. I thought I was going to be able to hook you up with a custom build tonight, but I've run into a big issue, which I raised with the relevant authorities. Even my patch would only work on Windows, for now, we're behind on the dependencies on other platforms.

What OS and python version are you on, out of curiousity?

# # Starbuck5 commented at 2022-07-24 08:57:41

@prajwalgatti If you need this feature right now, I've put up a pull request implementing it (only on Windows though, because dependencies)

You can download a wheel from the artifacts tab of one of these runs: https://ci.appveyor.com/project/pygame/pygame/builds/44257337

If you're on Mac or Linux you'll have to build off my branch and update the dependencies locally to SDL_ttf 2.20.0.

# # prajwalgatti commented at 2022-07-26 07:41:59

@Starbuck5 Sorry I just saw your replies today. Thanks a ton for the help! :)
I'm on a Linux machine and use python3.7. Will try to get my hands on a Windows machine, if not I'll do as you suggested for the Linux build.

Will update here after trying!

The text was updated successfully, but these errors were encountered:

zoldalma999 · 2023-07-23T21:47:36Z

Both manylinux and mac now build with harfbuzz, and the mentioned functions are also implemented. Closing as completed.

GalacticEmperor1 added font pygame.font hard labels Feb 12, 2023

MightyJosip removed hard labels Feb 15, 2023

zoldalma999 closed this as completed Jul 23, 2023

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Indic Scripts Including Hindi, Telugu, Tamil and many more not rendering as they supposed to be. (464) #431

Indic Scripts Including Hindi, Telugu, Tamil and many more not rendering as they supposed to be. (464) #431

GalacticEmperor1 commented Feb 12, 2023

zoldalma999 commented Jul 23, 2023