Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added indian tables by sreeja.param@gmail.com #3

Merged
merged 12 commits into from Sep 1, 2014
Merged

Conversation

egli
Copy link
Member

@egli egli commented May 28, 2014

These tables need some improvements before they can go in:

  • Integration with the Makefile.am (I can take care of that)
  • add an entry to the NEWS file about the changes/improvements
  • proper copyright statements and license headers
  • update of the AUTHORS file with all the authors
  • when changing existing tables then retain the existing copyright notices and license headers (unless your table is a complete rewrite.
  • Fix make check
  • Delete the old tables

These tables need some improvements before they can go in:
- Integration with Makefile
- Adapt AUTHORS file
- add proper copyright and license headers
- if they are based on previous tables then retain the copyright and the
  license
@egli egli added this to the 2.6.0 milestone May 28, 2014
@egli
Copy link
Member Author

egli commented Jun 2, 2014

OK, the copyright and the license should be according to this post in the mailing list.

@egli
Copy link
Member Author

egli commented Jun 2, 2014

I think I retained all the existing copyrights when integrating the new tables

@egli
Copy link
Member Author

egli commented Jun 12, 2014

make check doesn't pass. The problem is that both devanagari.cti and bengali.cti define the class halal. Now if you include both devanagari.cti and bengali.cti which is done in new.ctb and sat.ctb, then you get a double definition of the character class halal. Liblouis consequently barfs with ../tables/bengali.cti:191: error: character class already defined.

@egli
Copy link
Member Author

egli commented Jun 26, 2014

So sreeja says that his tables supersede all the tables originally provided by Samuel Thibault. That would mean all of

  • tables/as.ctb
  • tables/awa.ctb
  • tables/bengali.cti
  • tables/bh.ctb
  • tables/bn.ctb
  • tables/bra.ctb
  • tables/devanagari.cti
  • tables/dra.ctb
  • tables/gon.ctb
  • tables/gu.ctb
  • tables/gujarati.cti
  • tables/gurmukhi.cti
  • tables/hi.ctb
  • tables/kannada.cti
  • tables/kha.ctb
  • tables/kn.ctb
  • tables/kok.ctb
  • tables/kru.ctb
  • tables/malayalam.cti
  • tables/ml.ctb
  • tables/mni.ctb
  • tables/mr.ctb
  • tables/mun.ctb
  • tables/mwr.ctb
  • tables/ne.ctb
  • tables/new.ctb
  • tables/or.ctb
  • tables/oriya.cti
  • tables/pa.ctb
  • tables/pi.ctb
  • tables/sa.ctb
  • tables/sat.ctb
  • tables/sd.ctb
  • tables/ta.ctb
  • tables/tamil.cti
  • tables/te.ctb
  • tables/telugu.cti

Need to confirm with sreeja

@egli
Copy link
Member Author

egli commented Aug 5, 2014

I say:

It has not been merged with master. My problem is that I don't know what to do with the old tables. There might be existing users out there that depend on them. So I'm reluctant to remove them. But if I don't remove them (or at least two of them) your tables will not work. So I'm in a bit of a pickle, you see.
Maybe we should move the tables into a subdirectory named"deprecated" and keep them for a couple of releases. We should try to stay backwards compatible.

Mesar says:

I am not aware that anyone is using them, and the point of these updated tables is that they are hopefully better, because they have had user testing. I will be happy to drop the old tables assuming we get harness tests for each touched table.

having them in a subdirectory will only complicate things i.e. people will expect things to work, but the paths will not be picked up. If we drop them, people or we can just download a old version of
liblouis, or go back in the git history.
In any case, having tests will ensure that the tables only improve in quality.

@egli
Copy link
Member Author

egli commented Aug 5, 2014

OK, so what I'll do is to simply delete the old tables and replace them with the new ones

| old table | Language        | new table    |
|-----------+-----------------+--------------|
| as        | Assamese        | as-in-g1.utb |
| bn        | Bengali         | be-in-g1.utb |
| gu        | Gujarati        | gu-in-g1.utb |
| gurmukhi  | Gurumukhi       | gurumuki.cti |
| hi        | Hindi           | hi-in-g1.utb |
| kn        | Kannada         | ka-in-g1.utb |
| ml        | Malayalam       | ml-in-g1.utb |
| mni       | Manipuri        | mn-in-g1.utb |
| mr        | Marathi         | mr-in-g1.utb |
| ne        | Nepali          | np-in-g1.utb |
| or        | Oriya           | or-in-g1.utb |
| pa        | Panjabi/Punjabi | pu-in-g1.utb |
| sa        | Sanskrit        | sa-in-g1.utb |
| sd        | Sindhi          | si-in-g1.utb |
| te        | Telugu          | te-in-g1.utb |

There are still more to go
@egli
Copy link
Member Author

egli commented Aug 5, 2014

OK, got rid of 15 of the old tables. Now I still have the following to go:

Table Language
awa.ctb Awadhi
bh.ctb Bihari
bra.ctb Braj
dra.ctb Dravidian
gon.ctb Gondi
kha.ctb Khasi
kok.ctb Konkani
kru.ctb Kurukh
mun.ctb Munda
mwr.ctb Marwari
new.ctb Old Newari
pi.ctb Pali
sat.ctb Santali
ta.ctb Tamil
tamil.cti Tamil

Is there a replacement for these tables, are they obsolete or should I just leave them?

@egli
Copy link
Member Author

egli commented Aug 15, 2014

OK, thanks to sreeja we are three tables down. Now we have left the following:

Table Language
bh.ctb Bihari
dra.ctb Dravidian
gon.ctb Gondi
kok.ctb Konkani
kru.ctb Kurukh
mun.ctb Munda
mwr.ctb Marwari
new.ctb Old Newari
pi.ctb Pali
sat.ctb Santali
ta.ctb Tamil
tamil.cti Tamil

@egli
Copy link
Member Author

egli commented Aug 15, 2014

sreeja says:

I am attaching the files that i can give as an replacement for the list that you have given. Rest you can keep it as it will not disturb our files.

i am attaching files for awadhi, khasi and braj language that you can replace with samuel files. For tamil, ta.ctb and tamil.cti is already there in my package.

@egli egli modified the milestones: 2.6.0, 2.7.0 Aug 26, 2014
@egli
Copy link
Member Author

egli commented Sep 1, 2014

After talking to Sreeja and Dipendra we decided to remove the tables that "cause" the conflict with Halant, namely new.ctb and sat.ctb. According to Dipendra they are not in use much.

These tables aren't in much use according to Dipendra Manocha and they
conflict with the new Indian tables.
@egli
Copy link
Member Author

egli commented Sep 1, 2014

Now the tests pass. We still have the following old tables. We'll just leave them for now

Table Language
bh.ctb Bihari
dra.ctb Dravidian
gon.ctb Gondi
kok.ctb Konkani
kru.ctb Kurukh
mun.ctb Munda
mwr.ctb Marwari
pi.ctb Pali
ta.ctb Tamil
tamil.cti Tamil

@egli egli merged commit 704de6b into master Sep 1, 2014
egli added a commit that referenced this pull request Sep 1, 2014
@mhameed
Copy link
Contributor

mhameed commented Sep 1, 2014

Hi Christian,

The tamil tables, we need to be careful that it doesn't break any
existing functionality.
The tamil tables were worked through with the NVDA tamil translator and
a school in the region.

I don't think we have a lot of tamil tests at the moment, but hopefully
this can be addressed in an upcoming release.
Never the less, please check that they dont break the existing tamil tests.

thanks,
Mesar

@egli
Copy link
Member Author

egli commented Sep 1, 2014

Oh, I just merged it right now. Let me check.

@egli
Copy link
Member Author

egli commented Sep 1, 2014

@mhameed I didn't even realize that we had a test harness for Tamil

@egli
Copy link
Member Author

egli commented Sep 1, 2014

@mhameed
Ran 4690 tests (15.37% success), with 3969 failures and 0 errors
This just sucks
Let me see how it was before

@egli
Copy link
Member Author

egli commented Sep 1, 2014

Ah, before we had a much better result
Ran 4690 tests (99.87% success), with 6 failures and 0 errors.

@mhameed
Copy link
Contributor

mhameed commented Sep 1, 2014

On Mon 01/09/14,02:23, Christian Egli wrote:

@mhameed
Ran 4690 tests (15.37% success), with 3969 failures and 0 errors
This just sucks

yes, thats why we should be careful.
So please revert anything to do with tamil.
We should not have one big pull request, but many smaller once, one per
language.
It would be more managable.

Mesar


Reply to this email directly or view it on GitHub:
#3 (comment)

@egli egli deleted the tables/indian branch June 25, 2015 09:20
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 25, 2022
Otherwise we get

/home/samy/brl/translation/liblouis-git/tests/tables/bad.ctb:1: error: opcode 'bad' not defined.
1 errors found.
tests/tables/bad.ctb could not be compiled

=================================================================
==3022215==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 61 byte(s) in 1 object(s) allocated from:
    #0 0x7f67684e0545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f676831b31a in __GI___strdup string/strdup.c:42
    liblouis#2 0x7f676849d47e in compileFile /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4650
    liblouis#3 0x7f676849d9b1 in compileTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4767
    liblouis#4 0x7f676849e011 in getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4939
    liblouis#5 0x7f676849dcb0 in _lou_getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4848
    liblouis#6 0x7f676849dd0e in lou_getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4860
    liblouis#7 0x7f676849e165 in lou_checkTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4978
    liblouis#8 0x55e477d791c2 in main /home/samy/brl/translation/liblouis-git/tests/checkTable.c:29
    liblouis#9 0x7f67682b47fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#10 0x55e477d79099 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/checkTable+0x1099)

SUMMARY: LeakSanitizer: 61 byte(s) leaked in 1 allocation(s).
FAIL checkTable (exit status: 23)
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 25, 2022
Since the "for" loop would always make "matches" NULL.

Otherwise we get in tests/findTable.log:

[...]
2 matches found

=================================================================
==3029162==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b636cc in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:58
    liblouis#2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 32 byte(s) in 2 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b65293 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#2 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#3 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#4 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b637f9 in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:84
    liblouis#2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

SUMMARY: LeakSanitizer: 80 byte(s) leaked in 4 allocation(s).
FAIL findTable (exit status: 23)
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 25, 2022
Since the "for" loop would always make "matches" NULL.

Otherwise we get in tests/findTable.log:

[...]
2 matches found

=================================================================
==3029162==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b636cc in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:58
    liblouis#2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 32 byte(s) in 2 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b65293 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#2 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#3 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#4 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b637f9 in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:84
    liblouis#2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

SUMMARY: LeakSanitizer: 80 byte(s) leaked in 4 allocation(s).
FAIL findTable (exit status: 23)
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 25, 2022
Since the "for" loop would always make "matches" NULL.

Otherwise we get in tests/findTable.log:

[...]
2 matches found

=================================================================
==3029162==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b636cc in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:58
    liblouis#2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 32 byte(s) in 2 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b65293 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#2 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#3 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#4 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f6433b637f9 in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:84
    liblouis#2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    liblouis#3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    liblouis#4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

SUMMARY: LeakSanitizer: 80 byte(s) leaked in 4 allocation(s).
FAIL findTable (exit status: 23)
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 25, 2022
Otherwise we get

/home/samy/brl/translation/liblouis-git/tests/tables/bad.ctb:1: error: opcode 'bad' not defined.
1 errors found.
tests/tables/bad.ctb could not be compiled

=================================================================
==3022215==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 61 byte(s) in 1 object(s) allocated from:
    #0 0x7f67684e0545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    liblouis#1 0x7f676831b31a in __GI___strdup string/strdup.c:42
    liblouis#2 0x7f676849d47e in compileFile /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4650
    liblouis#3 0x7f676849d9b1 in compileTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4767
    liblouis#4 0x7f676849e011 in getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4939
    liblouis#5 0x7f676849dcb0 in _lou_getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4848
    liblouis#6 0x7f676849dd0e in lou_getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4860
    liblouis#7 0x7f676849e165 in lou_checkTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4978
    liblouis#8 0x55e477d791c2 in main /home/samy/brl/translation/liblouis-git/tests/checkTable.c:29
    liblouis#9 0x7f67682b47fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#10 0x55e477d79099 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/checkTable+0x1099)

SUMMARY: LeakSanitizer: 61 byte(s) leaked in 1 allocation(s).
FAIL checkTable (exit status: 23)
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 26, 2022
SwapTest has to check wheter it overflows the input. Otherwise we get in
tests/braille-specs/da-dk-g16-lit.log:

==43588==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000026712 at pc 0x7f7d9ff57381 bp 0x7fff050fd170 sp 0x7fff050fd168
READ of size 2 at 0x602000026712 thread T0
    #0 0x7f7d9ff57380 in swapTest /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:399
    liblouis#1 0x7f7d9ff5d22e in passDoTest /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:861
    liblouis#2 0x7f7d9ff552f2 in findForPassRule /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:211
    liblouis#3 0x7f7d9ff55ac9 in makeCorrections /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:276
    liblouis#4 0x7f7d9ff6128d in _lou_translate /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:1272
    liblouis#5 0x55b837c9bdac in check_base /home/samy/brl/translation/liblouis-git/tools/brl_checks.c:180
    liblouis#6 0x55b837c97717 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:807
    liblouis#7 0x55b837c97f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    liblouis#8 0x55b837c9948a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    liblouis#9 0x7f7d9fd447fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#10 0x55b837c914c9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tools/.libs/lou_checkyaml+0x64c9)

0x602000026712 is located 0 bytes to the right of 2-byte region [0x602000026710,0x602000026712)
allocated by thread T0 here:
    #0 0x7f7da00777cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    liblouis#1 0x55b837c9b8b6 in check_base /home/samy/brl/translation/liblouis-git/tools/brl_checks.c:138
    liblouis#2 0x55b837c97717 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:807
    liblouis#3 0x55b837c97f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    liblouis#4 0x55b837c9948a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    liblouis#5 0x7f7d9fd447fc in __libc_start_main ../csu/libc-start.c:332
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 26, 2022
read_test gives parsed_strlen(word) to read_options, so that it gives it
to read_typeforms, and thus typeform has that length. check_base should
thus also use that length when memcpying it.

Otherwise we get in braille-specs/de-g1.yaml:

==95136==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000f451c at pc 0x7f5603f1b9b0 bp 0x7ffe24c6bc60 sp 0x7ffe24c6b410
READ of size 16 at 0x6020000f451c thread T0
    #0 0x7f5603f1b9af in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
    liblouis#1 0x5626d22d983b in check_base /home/samy/brl/translation/liblouis-git/tools/brl_checks.c:116
    liblouis#2 0x5626d22d5717 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:807
    liblouis#3 0x5626d22d5f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    liblouis#4 0x5626d22d748a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    liblouis#5 0x7f5603c5d7fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#6 0x5626d22cf4c9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tools/.libs/lou_checkyaml+0x64c9)

0x6020000f451c is located 0 bytes to the right of 12-byte region [0x6020000f4510,0x6020000f451c)
allocated by thread T0 here:
    #0 0x7f5603f90987 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154
    liblouis#1 0x5626d22d37b5 in read_typeforms /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:567
    liblouis#2 0x5626d22d40c6 in read_options /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:642
    liblouis#3 0x5626d22d51e7 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:774
    liblouis#4 0x5626d22d5f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    liblouis#5 0x5626d22d748a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    liblouis#6 0x7f5603c5d7fc in __libc_start_main ../csu/libc-start.c:332
sthibaul added a commit to sthibaul/liblouis that referenced this pull request Mar 26, 2022
read_test gives parsed_strlen(word) to read_options, so that it gives it
to read_typeforms, and thus typeform has that length. check_base should
thus also use that length when memcpying it.

Otherwise we get in braille-specs/de-g1.yaml:

==95136==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000f451c at pc 0x7f5603f1b9b0 bp 0x7ffe24c6bc60 sp 0x7ffe24c6b410
READ of size 16 at 0x6020000f451c thread T0
    #0 0x7f5603f1b9af in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
    liblouis#1 0x5626d22d983b in check_base /home/samy/brl/translation/liblouis-git/tools/brl_checks.c:116
    liblouis#2 0x5626d22d5717 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:807
    liblouis#3 0x5626d22d5f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    liblouis#4 0x5626d22d748a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    liblouis#5 0x7f5603c5d7fc in __libc_start_main ../csu/libc-start.c:332
    liblouis#6 0x5626d22cf4c9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tools/.libs/lou_checkyaml+0x64c9)

0x6020000f451c is located 0 bytes to the right of 12-byte region [0x6020000f4510,0x6020000f451c)
allocated by thread T0 here:
    #0 0x7f5603f90987 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154
    liblouis#1 0x5626d22d37b5 in read_typeforms /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:567
    liblouis#2 0x5626d22d40c6 in read_options /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:642
    liblouis#3 0x5626d22d51e7 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:774
    liblouis#4 0x5626d22d5f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    liblouis#5 0x5626d22d748a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    liblouis#6 0x7f5603c5d7fc in __libc_start_main ../csu/libc-start.c:332
bertfrees pushed a commit that referenced this pull request Mar 28, 2022
Otherwise we get

/home/samy/brl/translation/liblouis-git/tests/tables/bad.ctb:1: error: opcode 'bad' not defined.
1 errors found.
tests/tables/bad.ctb could not be compiled

=================================================================
==3022215==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 61 byte(s) in 1 object(s) allocated from:
    #0 0x7f67684e0545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    #1 0x7f676831b31a in __GI___strdup string/strdup.c:42
    #2 0x7f676849d47e in compileFile /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4650
    #3 0x7f676849d9b1 in compileTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4767
    #4 0x7f676849e011 in getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4939
    #5 0x7f676849dcb0 in _lou_getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4848
    #6 0x7f676849dd0e in lou_getTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4860
    #7 0x7f676849e165 in lou_checkTable /home/samy/brl/translation/liblouis-git/liblouis/compileTranslationTable.c:4978
    #8 0x55e477d791c2 in main /home/samy/brl/translation/liblouis-git/tests/checkTable.c:29
    #9 0x7f67682b47fc in __libc_start_main ../csu/libc-start.c:332
    #10 0x55e477d79099 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/checkTable+0x1099)

SUMMARY: LeakSanitizer: 61 byte(s) leaked in 1 allocation(s).
FAIL checkTable (exit status: 23)
bertfrees pushed a commit that referenced this pull request Mar 28, 2022
Since the "for" loop would always make "matches" NULL.

Otherwise we get in tests/findTable.log:

[...]
2 matches found

=================================================================
==3029162==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    #1 0x7f6433b636cc in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:58
    #2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    #3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    #4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    #5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 32 byte(s) in 2 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    #1 0x7f6433b65293 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    #2 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    #3 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    #4 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

Indirect leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f6433b92545 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:75
    #1 0x7f6433b637f9 in list_conj /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:84
    #2 0x7f6433b652c7 in lou_findTables /home/samy/brl/translation/liblouis-git/liblouis/metadata.c:767
    #3 0x55c2bddf4291 in main /home/samy/brl/translation/liblouis-git/tests/findTable.c:33
    #4 0x7f64339667fc in __libc_start_main ../csu/libc-start.c:332
    #5 0x55c2bddf40d9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tests/.libs/findTable+0x10d9)

SUMMARY: LeakSanitizer: 80 byte(s) leaked in 4 allocation(s).
FAIL findTable (exit status: 23)
bertfrees pushed a commit that referenced this pull request Mar 28, 2022
SwapTest has to check wheter it overflows the input. Otherwise we get in
tests/braille-specs/da-dk-g16-lit.log:

==43588==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000026712 at pc 0x7f7d9ff57381 bp 0x7fff050fd170 sp 0x7fff050fd168
READ of size 2 at 0x602000026712 thread T0
    #0 0x7f7d9ff57380 in swapTest /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:399
    #1 0x7f7d9ff5d22e in passDoTest /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:861
    #2 0x7f7d9ff552f2 in findForPassRule /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:211
    #3 0x7f7d9ff55ac9 in makeCorrections /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:276
    #4 0x7f7d9ff6128d in _lou_translate /home/samy/brl/translation/liblouis-git/liblouis/lou_translateString.c:1272
    #5 0x55b837c9bdac in check_base /home/samy/brl/translation/liblouis-git/tools/brl_checks.c:180
    #6 0x55b837c97717 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:807
    #7 0x55b837c97f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    #8 0x55b837c9948a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    #9 0x7f7d9fd447fc in __libc_start_main ../csu/libc-start.c:332
    #10 0x55b837c914c9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tools/.libs/lou_checkyaml+0x64c9)

0x602000026712 is located 0 bytes to the right of 2-byte region [0x602000026710,0x602000026712)
allocated by thread T0 here:
    #0 0x7f7da00777cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x55b837c9b8b6 in check_base /home/samy/brl/translation/liblouis-git/tools/brl_checks.c:138
    #2 0x55b837c97717 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:807
    #3 0x55b837c97f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    #4 0x55b837c9948a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    #5 0x7f7d9fd447fc in __libc_start_main ../csu/libc-start.c:332
bertfrees pushed a commit that referenced this pull request Mar 28, 2022
read_test gives parsed_strlen(word) to read_options, so that it gives it
to read_typeforms, and thus typeform has that length. check_base should
thus also use that length when memcpying it.

Otherwise we get in braille-specs/de-g1.yaml:

==95136==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000f451c at pc 0x7f5603f1b9b0 bp 0x7ffe24c6bc60 sp 0x7ffe24c6b410
READ of size 16 at 0x6020000f451c thread T0
    #0 0x7f5603f1b9af in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
    #1 0x5626d22d983b in check_base /home/samy/brl/translation/liblouis-git/tools/brl_checks.c:116
    #2 0x5626d22d5717 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:807
    #3 0x5626d22d5f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    #4 0x5626d22d748a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    #5 0x7f5603c5d7fc in __libc_start_main ../csu/libc-start.c:332
    #6 0x5626d22cf4c9 in _start (/home/samy/ens/projet/1/translation/liblouis-git/tools/.libs/lou_checkyaml+0x64c9)

0x6020000f451c is located 0 bytes to the right of 12-byte region [0x6020000f4510,0x6020000f451c)
allocated by thread T0 here:
    #0 0x7f5603f90987 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:154
    #1 0x5626d22d37b5 in read_typeforms /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:567
    #2 0x5626d22d40c6 in read_options /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:642
    #3 0x5626d22d51e7 in read_test /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:774
    #4 0x5626d22d5f69 in read_tests /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:869
    #5 0x5626d22d748a in main /home/samy/brl/translation/liblouis-git/tools/lou_checkyaml.c:1063
    #6 0x7f5603c5d7fc in __libc_start_main ../csu/libc-start.c:332
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants