Skip to content
This repository has been archived by the owner. It is now read-only.
Browse files
Merge pull request #3 from fhieber/packing
updated packedGrammar documentation for 6.0.5 release
  • Loading branch information
mjpost committed Oct 22, 2015
2 parents b403216 + 0652f03 commit 624ca35ff155a7f2ef14e45e441d8d5eb4994f2e
Showing 1 changed file with 26 additions and 5 deletions.
@@ -18,20 +18,20 @@ The script can be found at
example usage. You can then add it to a Joshua config file, simply
replacing a `tm` path to the compressed text-file format with a path
to the packed grammar directory (Joshua will automatically detect that
it is packed.
it is packed, since a packed grammar is a directory).

Packing the grammar requires first sorting it, which can take quite a
bit of temporary space.
Packing the grammar requires first sorting it by the rules source side,
which can take quite a bit of temporary space.

*CAVEAT*: You may run into problems packing very large hiero
*CAVEAT*: You may run into problems packing very very large Hiero
grammars. Email the support list if you do.

### Examples

A Hiero grammar, using the compressed text file version:

tm = hiero -owner pt -maxspan 20 -path grammar.filtered.gz

Pack it:

$JOSHUA/scripts/support/ grammar.filtered.gz grammar.packed
@@ -51,3 +51,24 @@ You have a different temp file location:
Update the config file line:

tm = hiero -owner pt -maxspan 20 -path grammar.packed

### Using multiple packed grammars (Joshua 6.0.5)

Packed grammars serialize their vocabularies which prevented the use of multiple
packed grammars during decoding. With Joshua 6.0.5, it is possible to use multiple packed grammars during decoding if they have the same serialized vocabulary.
This is achieved by packing these grammars jointly using a revised packing CLI.

To pack multiple grammars:

$JOSHUA/scripts/support/ grammar1.filtered.gz grammar2.filtered.gz [...] grammar1.packed grammar2.packed [...]

This will produce two packed grammars with the same vocabulary. To use them in the decoder, put this in your ```joshua.config```:

tm = hiero -owner pt -maxspan 20 -path grammar1.packed
tm = hiero -owner pt2 -maxspan 20 -path grammar2.packed

Note the different owners.
If you are trying to load multiple packed grammars that do not have the same
vocabulary, the decoder will throw a RuntimeException at loading time:

Exception in thread "main" java.lang.RuntimeException: Trying to load multiple packed grammars with different vocabularies! Have you packed them jointly?

0 comments on commit 624ca35

Please sign in to comment.