Languages with duplicate phonemes

[drammock]

There are some languages that are showing duplicate phonemes. Not sure if this is caused by the denormRenorm function, or if there are errors in the source data files, or some other cause. Affects 19 languages in SAPHON, 10 languages in AA, 3 languages in RA.

[bambooforest]

Can you provide me with an example or two, then I'll go through and check.

On Wed, Oct 1, 2014 at 12:43 PM, Daniel McCloy notifications@github.com
wrote:

There are some languages that are showing duplicate phonemes. Not sure if
this is caused by the denormRenorm function, or if there are errors in
the source data files, or some other cause. Affects 19 languages in SAPHON,
10 languages in AA, 3 languages in RA.

—
Reply to this email directly or view it on GitHub
#53.

[drammock]

Writing from phone so can't do that easily. If you run the aggregation
script and then run the test script in this PR it will spit out the ISO
codes of the problem languages. Also this snippet may be useful:

foo <- split(final.data, final.data$LanguageCode)
sapply(foo, function(x) length(unique(x$Phonemes)) < length(x$Phonemes))

On Oct 1, 2014 6:48 PM, "Steven Moran" notifications@github.com wrote:

Can you provide me with an example or two, then I'll go through and check.

On Wed, Oct 1, 2014 at 12:43 PM, Daniel McCloy notifications@github.com
wrote:

There are some languages that are showing duplicate phonemes. Not sure
if
this is caused by the denormRenorm function, or if there are errors in
the source data files, or some other cause. Affects 19 languages in
SAPHON,
10 languages in AA, 3 languages in RA.

—
Reply to this email directly or view it on GitHub
#53.

—
Reply to this email directly or view it on GitHub
#53 (comment).

[drammock]

On further inspection these errors seem to be of two types:

1. cases where a single resource (aa, ra, saphon) has data on two different doculects that are classified with the same ISO code. In such cases we get lots of duplicate phonemes (basically, the set intersection of the inventories of the two dialects).
2. cases where something else is going wrong, and the allophone collapsing code doesn't reduce properly.

[drammock]

Many of these errors have been resolved, but #64 blocks this.