Rework release scripts for v1.4 #16

goodmami · 2021-10-27T22:36:58Z

This changes this quite a bit for building the wordnets.

There's an index.toml file which contains metadata (language, citation, etc.) for each wordnet, as well as pointers to the source (.tab) files.
The script scripts/tsv2lmf.py is no longer callable directly.
The scripts/ directory is now a Python package so relative imports work.
Need to install dependencies (pip install -r --requirements.txt) and get required files.
Call the conversion script with python -m scripts.build --version=XYZ [id...]
Built wordnets go in build/omw-XYZ/... (not tracked)
Compressed packages go in release/... (not tracked)
Several files have been removed (scripts/ili-map.tab, scripts/WN-LMF.dtd, wns/omw-citations.tab) because they are unnecessary or can be retrieved during a build.
Resulting wordnets follow WN-LMF-1.1.dtd

There are still some things to do with the GitHub Actions workflow:

Figure out how building wn30 and wn31 with scripts/wndb2lmf fits
download required files
build the wordnets with the new scripts
package the build for release

At least for the codepoints in html.entities.codepoint2name and the custom ones, there are no names that are valid 4-digit hexadecimal values, so we don't need something like 0x prefixing the codepoint.

goodmami · 2021-10-29T22:41:51Z

The latest changes address 3 of 4 of the tasks listed above. There are some new ones, though:

fcbond · 2021-10-30T09:10:09Z

I think the only remaining issues are the two above and the availability of wn 0.8.1

goodmami · 2021-11-02T04:49:32Z

Wn 0.8.1 had a packaging issue so I released Wn 0.8.2, however it looks like I might not have caught some LMF export issues, so I'm investigating before releasing 0.8.3.

Regarding WordNet 3.0, my validation script found some other issues caused by issues in the source data (shown below, abbreviated):

Nine redundant relations

$ grep 'benignancy' index.noun 
benignancy n 1 4 ! @ = + 1 0 04840981  
$ grep '^04840981' data.noun 
04840981 07 n 03 benignity 0 benignancy 0 ... + 01372773 a 0201 + 01372773 a 0201 ... + 01372773 a 0101 ...
$ #                                           ^ here            ^ same                ^ different source sense

One self-loop

$ grep 'unicycle' index.noun 
unicycle n 1 2 @ + 1 0 04509417  
$ grep '^04509417' data.noun 
04509417 06 n 02 unicycle 0 monocycle 0 ... + 04509417 n 0101 ...
$ #                                         ^ points to self

Many, many relations without the reverse relations, mostly pertainym, also is_entailed_by, is_caused_by, and also. This should probably be ignored, as that's how the data was designed.

I can fix (1) by simply dropping the redundant relations. (2) may require a manual fix, as we did for inhibit.

goodmami · 2021-11-02T04:56:37Z

(2) may require a manual fix, as we did for inhibit.

Or maybe we just drop that one, too. That synset already has derivation links to unicyclist and unicycle (v).

fcbond · 2021-11-02T05:01:54Z

But I would be very happy to drop this. Maybe check both for duplicate relations (and discard) and self loops (and discard)? I think there is also a loop through categories with computer and computer science, each being in the domain of the other --- but I am not sure this is really a bug. In OMW 2.0 I only checked for self-loops and I only cycles with hypernyms (+ instances). I would expect loops with derivations (although the self loop seems silly).

…

On Tue, Nov 2, 2021 at 12:56 PM Michael Wayne Goodman < ***@***.***> wrote: (2) may require a manual fix, as we did for *inhibit*. Or maybe we just drop that one, too. That synset already has derivation links to *unicyclist* and *unicycle* (v). — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIPZRSHNPOYM3CRVS2FVJLUJ5VRBANCNFSM5G3OFFZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Francis Bond <http://www3.ntu.edu.sg/home/fcbond/> Division of Linguistics and Multilingual Studies Nanyang Technological University

goodmami · 2021-11-02T05:04:36Z

Ok sounds good. Also, WordNet 3.1 has the same problems, plus it also has about 40 synsets with the same ILI as another synset. E.g.:

Synset ID	ILI
omw-en31-00767587-n	i39442
omw-en31-00767761-n	i39442
omw-en31-00780744-n	i39502
omw-en31-00781071-n	i39502
...	...

fcbond · 2021-11-02T05:11:33Z

Wow! I have not looked at PWN 3.1 but this looks like a bug in the mapping. @jmccrae do you know anything about this? I guess we have four options for this release: (i) randomly pick one (ii) not link either (and not somewhere) (iii) manually fix them (iv) hope John has a solution If (iv) does not happen then I would prefer to go with (ii).

…

On Tue, Nov 2, 2021 at 1:04 PM Michael Wayne Goodman < ***@***.***> wrote: Ok sounds good. Also, WordNet 3.1 has the same problems, plus it also has about 40 synsets with the same ILI as another synset. E.g.: Synset ID ILI omw-en31-00767587-n i39442 omw-en31-00767761-n i39442 omw-en31-00780744-n i39502 omw-en31-00781071-n i39502 ... ... — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIPZRSGIYODBXQZXCG6QGDUJ5WO5ANCNFSM5G3OFFZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Francis Bond <http://www3.ntu.edu.sg/home/fcbond/> Division of Linguistics and Multilingual Studies Nanyang Technological University

jmccrae · 2021-11-02T10:27:11Z

It seems that this is an error in the alignment script that was used to generate the WN 3.1 mapping.

It seems that some of the synsets that were new to PWN 3.1 have in some cases been assigned a key in that file that is incorrect. Instead there should be no assigned key. I will fix this in the CILI repo

jmccrae · 2021-11-02T11:13:10Z

Actually, if you look at Turtle version of the mapping you can see what the solution is:

ili:i39442 skos:closeMatch pwn31:100767587-n . # crime, offense, criminal_offense, criminal_offence, offence, law-breaking
ili:i39442 owl:sameAs pwn31:100767761-n . # crime, offense, criminal_offense, criminal_offence, offence, law-breaking

In all cases, one is marked as a close match and one as an exact match

goodmami · 2021-11-02T20:41:15Z

I think the solution is to remove the mappings in the .tab file for all skos:closeMatch links? I've opened globalwordnet/cili#13 to discuss.

goodmami · 2021-11-03T04:54:03Z

Maybe check both for duplicate
relations (and discard) and self loops (and discard)?

Returning to the above issue, I just compared the NLTK's wordnet files and the one's we're using, and they are currently identical (after we fixed the inhibit loop). That is, the NLTK also has the unicycle loop and redundant edges:

>>> from nltk.corpus import wordnet as wn
>>> wn.lemmas('unicycle', pos='n')
[Lemma('unicycle.n.01.unicycle')]
>>> wn.lemmas('unicycle', pos='n')[0].derivationally_related_forms()
[Lemma('unicyclist.n.01.unicyclist'), Lemma('unicycle.n.01.unicycle'), Lemma('unicycle.v.01.unicycle')]
>>> wn.lemmas('benignancy')[0].derivationally_related_forms()
[Lemma('benign.s.03.benign'), Lemma('benign.s.03.benign')]

Considering this, I think we should refrain from fixing further things, even if it's tempting. That's what PWN 3.1 and OEWN are for. We could document the known issues, though. An exception could be made if the data, due to those errors, is no longer loadable in Wn.

fcbond · 2021-11-03T05:29:13Z

OK, that makes sense. I will add something to the Readme that these issues exist, and that they are fixed in later versions.

…

On Wed, Nov 3, 2021 at 12:54 PM Michael Wayne Goodman < ***@***.***> wrote: Maybe check both for duplicate relations (and discard) and self loops (and discard)? Returning to the above issue, I just compared the NLTK's wordnet files and the one's we're using, and they are currently identical (after we fixed the *inhibit* loop). That is, the NLTK also has the unicycle loop and redundant edges: >>> from nltk.corpus import wordnet as wn >>> wn.lemmas('unicycle', pos='n') [Lemma('unicycle.n.01.unicycle')] >>> wn.lemmas('unicycle', pos='n')[0].derivationally_related_forms() [Lemma('unicyclist.n.01.unicyclist'), Lemma('unicycle.n.01.unicycle'), Lemma('unicycle.v.01.unicycle')] >>> wn.lemmas('benignancy')[0].derivationally_related_forms() [Lemma('benign.s.03.benign'), Lemma('benign.s.03.benign')] Considering this, I think we should refrain from fixing further things, even if it's tempting. That's what PWN 3.1 and OEWN are for. We could document the known issues, though. An exception could be made if the data, due to those errors, is no longer loadable in Wn. — You are receiving this because your review was requested. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIPZRUHHJXTTEKCQXSCRMLUKDFALANCNFSM5G3OFFZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Francis Bond <http://www3.ntu.edu.sg/home/fcbond/> Division of Linguistics and Multilingual Studies Nanyang Technological University

It points to the omw-data project page, which is not the URL for the resource, so it can't be used to automatically download the dependency.

Currently only README and LICENSE without extensions since we don't have README.md, LICENSE.txt, etc. in the data. It is trivial to enable these, though. Also, citation.bib is copied if present.

This way all XML file names are the same as their directories. The predictability makes other things easier.

Build script now requires a version argument. The release workflow should work either when the release is created or on a manual dispatch.

goodmami · 2021-11-04T00:01:36Z

@fcbond I did a bit more work on this; see the commit messages above. I think most things are done now, but the new workflow is untested. I added a "workflow_dispatch" event, so you can click a "Run workflow" button on GitHub to start the action. This might be helpful if there's a failure or we otherwise need to rerun the build. It still starts automatically when a release is created.

Anything else to do?

The omw-en and omw-en31 lexicons are now generated from WordNet sources directly, so the stored xml.xz files are no longer needed. This commit also repurposes, and renames, those files' directories for storing additional files related to the wordnets, such as the LICENSE, README.md, and citation.bib files. This commit also updates and expands the READMEs a bit.

goodmami · 2021-11-04T05:50:07Z

And a bit more work. I pushed the 0.8.3 version of Wn so that dependency is bumped here. I also removed the old pwn files and updated their READMEs, which are now copied into the built omw-en* directories.

I realize that we're not building the index.toml file for Wn anymore. But otherwise I think this is done.

fcbond · 2021-11-04T06:36:16Z

The release script currently includes both omw-en and omw-en31 in omw-1.4. I think we should probably only include omw-en. This is a difference from omw-1.3 which included neither, but as the new wordnets require it, I think it makes sense to bundle it together. I will push a fix. Should we build a new index.toml? Should it be part of build.sh?

…

On Thu, Nov 4, 2021 at 1:50 PM Michael Wayne Goodman < ***@***.***> wrote: And a bit more work. I pushed the 0.8.3 version of Wn so that dependency is bumped here. I also removed the old pwn files and updated their READMEs, which are now copied into the built omw-en* directories. I realize that we're not building the index.toml file for Wn anymore. But otherwise I think this is done. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIPZRROGAOW2PGABOO6APDUKIUKTANCNFSM5G3OFFZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Francis Bond <http://www3.ntu.edu.sg/home/fcbond/> Division of Linguistics and Multilingual Studies Nanyang Technological University

goodmami · 2021-11-04T16:02:58Z

Regarding not bundling omw-en31, that's a good idea. Maybe this is as easy as adding --exclude="omw-$VERSION/omw-en31/" to the tar command in .github/workflows/release.yml?

goodmami · 2021-11-04T16:04:40Z

Regarding index.toml, adding it to build.sh make senses. I'll think about it.

fcbond · 2021-11-04T16:15:40Z

I did that. Maybe I forgot to push

…

On Fri, 5 Nov 2021, 12:03 am Michael Wayne Goodman, < ***@***.***> wrote: Regarding not bundling omw-en31, that's a good idea. Maybe this is as easy as adding --exclude="omw-$VERSION/omw-en31/" to the tar command in .github/workflows/release.yml? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIPZRW4LANAKJUIGY6D3FTUKK4EZANCNFSM5G3OFFZA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

The workflow just calls the scripts. This makes it easier to test things out locally.

goodmami · 2021-11-04T23:06:25Z

Maybe I forgot to push

Yeah, I didn't see it. No worries, I pushed a commit which does this. It also moves most shell commands into local scripts so you can test things locally. Just be careful with the --publish option on package.sh, as it actually uploads things, but in practice I think it would fail unless you had the GITHUB_TOKEN environment variable appropriately set.

goodmami · 2021-11-04T23:58:11Z

Oh, and the index.toml file is now being built in the package.sh script.

Rework release scripts for v1.4

2468d06

goodmami requested a review from fcbond October 27, 2021 22:36

fcbond and others added 10 commits October 29, 2021 18:19

update labels and urls

f925a36

documentation for wordnet 3.0

bc986be

Fix a typo and a redundant url

7ad4bf7

Fix a bug wtih lexfile lookup

d3d325b

Make wndb2lmf progress reporting more clear

fa7371e

Use hex-codepoints for chars without known escape

6e2d09d

At least for the codepoints in html.entities.codepoint2name and the custom ones, there are no names that are valid 4-digit hexadecimal values, so we don't need something like 0x prefixing the codepoint.

Add build.sh and tracked etc/ files

1fa618a

Fix path to CILI files in tsv2lmf.py

72d8ad5

Use proper string formatting

4b31d2d

Bump Wn version to 0.8.1

36bde23

fcbond added 3 commits October 30, 2021 16:16

changed Albanian language code to sq; closes #18

5e35694

renamed packages to omw-LG; closes #15

53e03de

Made the README more readable

1445441

goodmami mentioned this pull request Nov 2, 2021

Extract wordnet into a separate package nltk/nltk#2423

Open

harmonized build directory names; closes #15 again

5999d07

First stab at a CITATION.cff

d256135

goodmami added 7 commits November 3, 2021 11:47

Code cleanup

f848ba0

Fix #17: separate sense_key and sense ID for omw-en

3c8a38a

Remove default url in requires

0d06a8f

It points to the omw-data project page, which is not the URL for the resource, so it can't be used to automatically download the dependency.

Copy extra files to LMF package, if present

9e59716

Currently only README and LICENSE without extensions since we don't have README.md, LICENSE.txt, etc. in the data. It is trivial to enable these, though. Also, citation.bib is copied if present.

Change WN3.0 build dir from omw-en30 to omw-en

5c06da1

This way all XML file names are the same as their directories. The predictability makes other things easier.

Add LICENSE and citation.bib files to omw-en* pkgs

adb1627

Update release workflow and build script

745da59

Build script now requires a version argument. The release workflow should work either when the release is created or on a manual dispatch.

goodmami added 5 commits November 3, 2021 17:09

Attempt fix of workflow file

7b2ed2b

Merge branch 'main' into omw-1.4

fbb6d99

Bump wn dependency version to 0.8.3

bfc90c3

Remove old make-lmf.bash script

9bc4e14

goodmami marked this pull request as ready for review November 4, 2021 05:50

Move most shell commands from workflow to scripts

67eebd4

The workflow just calls the scripts. This makes it easier to test things out locally.

Remove workflow_dispatch action option

76f60fb

goodmami merged commit 3393edf into main Nov 5, 2021

goodmami deleted the omw-1.4 branch November 5, 2021 03:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework release scripts for v1.4 #16

Rework release scripts for v1.4 #16

goodmami commented Oct 27, 2021 •

edited

goodmami commented Oct 29, 2021 •

edited

fcbond commented Oct 30, 2021

goodmami commented Nov 2, 2021

goodmami commented Nov 2, 2021

fcbond commented Nov 2, 2021 via email

goodmami commented Nov 2, 2021

fcbond commented Nov 2, 2021 via email

jmccrae commented Nov 2, 2021

jmccrae commented Nov 2, 2021

goodmami commented Nov 2, 2021

goodmami commented Nov 3, 2021

fcbond commented Nov 3, 2021 via email

goodmami commented Nov 4, 2021

goodmami commented Nov 4, 2021

fcbond commented Nov 4, 2021 via email

goodmami commented Nov 4, 2021

goodmami commented Nov 4, 2021

fcbond commented Nov 4, 2021 via email

goodmami commented Nov 4, 2021

goodmami commented Nov 4, 2021

Rework release scripts for v1.4 #16

Rework release scripts for v1.4 #16

Conversation

goodmami commented Oct 27, 2021 • edited

goodmami commented Oct 29, 2021 • edited

fcbond commented Oct 30, 2021

goodmami commented Nov 2, 2021

goodmami commented Nov 2, 2021

fcbond commented Nov 2, 2021 via email

goodmami commented Nov 2, 2021

fcbond commented Nov 2, 2021 via email

jmccrae commented Nov 2, 2021

jmccrae commented Nov 2, 2021

goodmami commented Nov 2, 2021

goodmami commented Nov 3, 2021

fcbond commented Nov 3, 2021 via email

goodmami commented Nov 4, 2021

goodmami commented Nov 4, 2021

fcbond commented Nov 4, 2021 via email

goodmami commented Nov 4, 2021

goodmami commented Nov 4, 2021

fcbond commented Nov 4, 2021 via email

goodmami commented Nov 4, 2021

goodmami commented Nov 4, 2021

goodmami commented Oct 27, 2021 •

edited

goodmami commented Oct 29, 2021 •

edited