Read from STDIN

[mjpost]

🚀 Feature

It would be really nice if COMET could read input from STDIN, e.g.,

# three fields triggers comet-ref
$ paste source.txt hyps.txt ref.txt | comet [args]

# two fields -> comet-src
$ paste source.txt hyps.txt | comet [args]

Motivation

This is consistent with standard UNIX usage. It is also slightly less cumbersome, and allows comet to be used in settings without writing files to disk.

[ricardorei]

I'll investigate this. It seems like a good feature.

Do you have support for this in sacrebleu?

If so I can start by looking into sacrebleu implementation

[mjpost]

Sacrebleu supports STDIN for the system output, but not for the ref (and doesn’t use the source). So the COMET-style use would be

$ cat system.txt | comet -s source.txt -r ref.txt

which is different from what I proposed.

But it gives me another idea, which would be to add support for sacrebleu-style builtin test sets, e.g.,

# one option
$ cat system.txt | comet -t wmt20 -l de-en [other args]

# another option
$ cat system.txt | comet --sacrebleu-testset wmt20/de-en
$ cat system.txt | comet --sacrebleu-testset mtedx/valid/pt-es

You could accomplish this by just using sacrebleu as a library. It’s pretty easy:

from sacrebleu.utils import get_source, get_references, get_files

# trigger sacrebleu test set
# make these optional: nargs=“?” for argparse
if args.source is None and args.references is None:
    if args.sacrebleu_dataset is None:
        # throw error

    # some test sets are hierarchical, e.g., “mtedx/valid”
    test_set, langpair = args.sacrebleu_dataset.rsplit(“/“, maxsplit=1)
    source = get_source(test_set, langpair)
    ref = get_referencees(test_set, langpair)

     # alternative
    source, ref, _ = get_files(test_set, langpair)

[mjpost]

(Note that, via @ozancaglayan, sacrebleu also supports tab-delimited system output on STDIN, and will then do significance testing among them, e.g.,

$ paste system1.txt system2.txt | sacrebleu -t wmt20 -l de-en

Just something else to consider in terms of CLI).

[ricardorei]

Thanks, Matt! These are nice features indeed! Do you want to submit a PR 😁 ?

If not I can still try to allocate some time to do them before the new release

[mjpost]

Hmm, good question. What’s your timeline for the release?

…

On Oct 19, 2021, at 15:30, Ricardo Rei ***@***.***> wrote:  Thanks, Matt! These are nice features indeed! Do you want to submit a PR 😁 ? If not I can still try to allocate some time to do them before the new release — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[ricardorei]

We were planning the release for the end of November beginning of December