Implementing a custom semantics parser on spacy 3.0

[DeltaSierra4]

tl;dr - Is the custom semantics parser not available on SpaCy 3.0 or am I missing a step in my data preparation steps below?
(Updated topic to reflect the actual question at hand)

Hello, I've been using the custom semantics parser that was introduced here on spacy 2.1. I tried to implement the same custom semantics parser on spacy 3.0 by converting my training set into .spacy format using Example.from_dict() with the relevant text converted into a Doc and annotations set like {'heads': [token indices], 'deps' [stuff like '-', '-', 'ROOT', '-', 'QUALITY']}. So the snippet of code looks like this:

trainset = [
    (
        "I received blue flowers from Jane",
        {
            "heads": [1, 1, 3, 2, 1, 1]
            "deps": ['-', 'ROOT', 'QUALITY', 'ITEM', '-', '-']
        }
    )
   ... <more data> ...
]

#More code here

docbin = DocBin()
for text, annotations in trainset:
    example = Example.from_dict(nlp.make_doc(text), annotations)
    docbin.add(example.reference)
docbin.to_disk("data/train.spacy")

When I run the spacy train script via CLI with the base config file I generated from python -m spacy init config config.cfg --pipeline 'parser' --gpu (didn't change any values except for the path to training and validation sets), I get the following error:

('D', 0, 9000.0)
('L-dep', 0, 9000.0)
('R--', 0, 9000.0)
('R-dep', 0, 9000.0)
('B-ROOT', 1, 1.0)
('Gold sent starts?', 1, 0)
('D', 0, 9000.0)
('L-dep', 0, 9000.0)
('R--', 0, 9000.0)
('R-dep', 0, 9000.0)
('B-ROOT', 1, 1.0)
('Gold sent starts?', 1, 0)
⚠ Aborting and saving the final best model. Encountered exception:
ValueError('Could not find gold transition - see logs above.')

Did I miss something? Is there something more that I need to provide for the dependency parser, or is custom semantics parsing no longer available on spacy 3.0? All and any help would be appreciated. Thank you in advance!

[adrianeboyd]

Hi, the default parser settings are set up for cases with (a lot) more data, so you can run into problems if there are just a few training examples.

Here I think the main culprit is the min_action_freq cutoff. Lowering it to 10 instead of 30 worked for me to get this example to work with an otherwise default spacy init config -p parser config. This still may not be the ideal setting for this particular toy example, but the training process could proceed.

This would probably be a useful example to convert to a new project, too.

[mehmetilker]

Hello @adrianeboyd,

I am having same problem above ("Could not find gold transition - see logs above.") and tried to change min_action_freq from 30 to 10.
But problem is different I guess.
I could train a model successfully with "UD_Turkish-IMST treebank" but using same config gave me following error with edited version of IMST (https://github.com/boun-tabi/UD_Turkish-BIMST)

Something is different in BIMST treebank I guess and training pipeline does not like it.
Is there a way to validate treebank data to see the problem?

Exception:

[2021-04-19 08:54:03,371] [INFO] Initialized pipeline components: ['transformer', 'tagger', 'morphologizer', 'lemmatizer', 'parser']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'tagger', 'morphologizer', 'lemmatizer',
'parser']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS TAGGER  LOSS MORPH...  LOSS PARSER  TAG_ACC  POS_ACC  MORPH_ACC  LEMMA_ACC  DEP_UAS  DEP_LAS  SENTS_F  SCORE 
---  ------  -------------  -----------  -------------  -----------  -------  -------  ---------  ---------  -------  -------  -------  ------
  0       0         347.70       226.53         232.76       685.27     3.15     9.21      36.66      80.80     9.99     9.99     0.02    0.29
('S', 0, 9000.0)
('D', 1, 2.0)
('L-nsubj', 1, 2.0)
('L-obl', 1, 2.0)
('L-nmod:poss', 1, 2.0)
('L-obj', 1, 2.0)
('L-punct', 1, 2.0)
('L-amod', 1, 2.0)
('L-advmod', 1, 1.0)
('L-det', 1, 2.0)
('L-nmod', 1, 2.0)
('L-acl', 1, 2.0)
('L-advcl', 1, 2.0)
('L-cc', 1, 2.0)
('L-conj', 1, 2.0)
('L-nummod', 1, 2.0)
('L-ccomp', 1, 2.0)
('L-iobj', 1, 2.0)
('L-appos', 1, 2.0)
('L-advmod:emph', 1, 2.0)
('L-advcl:cond', 1, 2.0)
('L-discourse', 1, 2.0)
('L-xcomp', 1, 2.0)
('L-csubj', 1, 2.0)
('L-fixed', 1, 2.0)
('L-case||advmod', 1, 2.0)
('L-conj||nsubj', 1, 2.0)
('L-punct||obj', 1, 2.0)
('L-conj||obj', 1, 2.0)
('L-conj||amod', 1, 2.0)
('L-compound', 1, 2.0)
('L-dep', 1, 2.0)
('R-punct', 1, 2.0)
('R-conj', 1, 2.0)
('R-compound', 1, 1.0)
('R-case', 1, 2.0)
('R-advmod:emph', 1, 2.0)
('R-flat', 1, 2.0)
('R-compound:lvc', 1, 2.0)
('R-nsubj', 1, 2.0)
('R-advmod', 1, 2.0)
('R-obl', 1, 2.0)
('R-compound:redup', 1, 2.0)
('R-aux:q', 1, 2.0)
('R-obj', 1, 2.0)
('R-cop', 1, 2.0)
('R-mark', 1, 2.0)
('R-nmod', 1, 2.0)
('R-punct||conj', 1, 2.0)
('R-discourse', 1, 2.0)
('R-appos', 1, 2.0)
('R-advcl', 1, 2.0)
('R-cc', 1, 2.0)
('R-conj||ccomp', 1, 2.0)
('R-iobj', 1, 2.0)
('R-cc||conj', 1, 2.0)
('R-amod', 1, 2.0)
('R-dep', 1, 2.0)
('B-ROOT', 0, 9000.0)
('Gold sent starts?', 0, 0)
⚠ Aborting and saving the final best model. Encountered exception:
ValueError('Could not find gold transition - see logs above.')
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/dist-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/spacy/cli/train.py", line 59, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 114, in train
    raise e
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 98, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/usr/local/lib/python3.7/dist-packages/spacy/training/loop.py", line 195, in train_while_improving
    subbatch, drop=dropout, losses=losses, sgd=False, exclude=exclude
  File "/usr/local/lib/python3.7/dist-packages/spacy/language.py", line 1107, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])
  File "spacy/pipeline/transition_parser.pyx", line 366, in spacy.pipeline.transition_parser.Parser.update
  File "spacy/pipeline/transition_parser.pyx", line 478, in spacy.pipeline.transition_parser.Parser.get_batch_loss
  File "spacy/pipeline/_parser_internals/arc_eager.pyx", line 824, in spacy.pipeline._parser_internals.arc_eager.ArcEager.set_costs
ValueError: Could not find gold transition - see logs above.

[adrianeboyd]

You can run some basic checks on the training data using debug data:

spacy debug data config.cfg --verbose

There might be a large number of non-projective arcs that lead to sparse training data? debug data should show warnings if there are low counts of particular (projectivized) dependency labels.

You probably wouldn't want to use this setting for the final model, but you can try min_action_freq = 1 to see if it's possible to train the model at all as in the teeny demo?

[mehmetilker]

Here is the result of debug data command:

✘ Found 2 projectivized train sentence(s) with cycles

I guess removing those sentences will solve the problem.
Is there any way to find them?

============================ Data file validation ============================
✔ Corpus is loadable
✔ Pipeline can be initialized with data

=============================== Training stats ===============================
Language: tr
Training pipeline: transformer, tagger, morphologizer, lemmatizer, parser
3685 training docs
975 evaluation docs
⚠ 7 training examples also in evaluation data

============================== Vocab & Vectors ==============================
ℹ 38083 total word(s) in the data (13781 unique)
⚠ 1101 misaligned tokens in the training data
⚠ 308 misaligned tokens in the dev data
10 most common words: '.' (3433), ',' (2324), 'bir' (736), 've' (353), 'de'
(284), 'ki' (279), 'da' (267), 'bu' (235), '"' (203), '...' (192)
ℹ No word vectors present in the package

=========================== Part-of-speech Tagging ===========================
ℹ 27 label(s) in data
'Noun' (9723), 'Verb' (6989), 'Punc' (6780), 'Adj' (2131), 'Conj' (1462), 'NAdj'
(1412), 'Adverb' (1403), 'Prop' (1391), 'ANum' (1141), 'Pers' (724), 'Det'
(632), 'PCNom' (515), 'Ques' (319), 'Demons' (227), 'PCDat' (199), 'NNum' (191),
'PCAbl' (131), 'Reflex' (120), 'Quant' (106), 'Abr' (85), 'Neg' (80), 'Interj'
(56), 'PCIns' (16), 'Dup' (15), 'Zero' (11), 'Postp' (5), 'PCAcc' (1)

============================= Dependency Parsing =============================
ℹ Found 3391 sentence(s) with an average length of 11.2 words.
⚠ The training data contains 0.92 sentences per document. When there
are very few documents containing more than one sentence, the parser will not
learn how to segment longer texts into sentences.
ℹ Found 302 nonprojective train sentence(s)
ℹ 39 label(s) in train data
ℹ 200 label(s) in projectivized train data
'punct' (6706), 'ROOT' (3383), 'nsubj' (2864), 'obl' (2604), 'nmod:poss' (2426),
'obj' (2384), 'conj' (2138), 'amod' (1802), 'advmod' (1743), 'det' (1251),
'compound' (1206), 'nmod' (1128), 'acl' (1016), 'case' (900), 'advmod:emph'
(641), 'advcl' (604), 'cc' (565), 'flat' (557), 'nummod' (359), 'compound:lvc'
(341), 'ccomp' (310), 'compound:redup' (143), 'iobj' (142), 'aux:q' (127),
'appos' (112), 'discourse' (86), 'cop' (79), 'mark' (57), 'advcl:cond' (56),
'xcomp' (39), 'csubj' (28), 'fixed' (25), 'dislocated' (14), 'orphan' (10),
'parataxis' (8), 'clf' (5), 'cc:preconj' (3), 'dep' (1), 'aux' (1)
⚠ Low number of examples for label 'dislocated' (14)
⚠ Low number of examples for label 'clf' (5)
⚠ Low number of examples for label 'parataxis' (8)
⚠ Low number of examples for label 'orphan' (10)
⚠ Low number of examples for label 'dep' (1)
⚠ Low number of examples for label 'cc:preconj' (3)
⚠ Low number of examples for label 'aux' (1)
⚠ Low number of examples for 160 label(s) in the projectivized
dependency trees used for training. You may want to projectivize labels such as
punct before training in order to improve parser performance.
⚠ Projectivized labels with low numbers of examples:
amod||nsubj: 5, punct||amod: 1, punct||acl: 6, dislocated||nsubj: 5, obj||csubj:
1, nmod||csubj: 1, nsubj||csubj: 1, cc||appos: 1, appos||nsubj: 6, advcl||advcl:
1, acl||nsubj: 6, nmod||mark: 1, conj||conj: 4, appos||obl: 1, appos||nmod: 2,
punct||appos: 6, advmod||appos: 2, nsubj||appos: 1, appos||obj: 2,
nmod:poss||nsubj: 10, obl||nsubj: 1, nsubj||advcl: 2, parataxis||case: 1,
nmod||conj: 7, punct||nmod:poss: 6, conj||iobj: 1, conj||amod: 10, nsubj||ccomp:
5, nsubj||nsubj: 4, case||advcl: 3, punct||case: 3, cc||ccomp: 1, conj||nsubj:
20, det||nsubj: 1, cc||conj: 15, amod||nmod:poss: 4, obl||advcl: 3, acl||obj: 3,
punct||csubj: 1, compound:redup||amod: 1, acl||conj: 2, punct||nsubj: 4,
conj||nmod: 6, case||nmod: 6, case||amod: 4, amod||nmod: 1, acl||nmod:poss: 2,
nummod||obj: 1, dislocated||obj: 6, nmod:poss||obl: 7, nummod||obl: 2,
nsubj||acl: 3, punct||cc: 1, nmod:poss||nmod: 3, compound||nmod:poss: 6,
nmod:poss||nmod:poss: 3, compound||nsubj: 3, advmod:emph||nsubj: 1, conj||obj:
14, conj||compound: 1, obl||obj: 6, punct||obj: 14, amod||conj: 5, amod||obl: 4,
obj||compound: 1, obl||compound: 1, flat||appos: 1, obl||conj: 9,
advmod:emph||cc: 1, nsubj||conj: 10, conj||nummod: 2, cc||nmod:poss: 1,
nsubj||obj: 4, advmod:emph||conj: 1, nmod:poss||obj: 6, nmod||nmod: 2,
conj||obl: 8, case||obl: 1, punct||obl: 2, nsubj||cop: 2, nummod||nmod:poss: 1,
acl||obl: 2, punct||nmod: 6, conj||acl: 1, conj||nmod:poss: 4, obj||advcl: 2,
obj||iobj: 1, obl||ccomp: 1, conj||appos: 1, ccomp||advcl: 2, case||advmod: 18,
orphan||nsubj: 1, case||orphan: 1, advmod||advcl: 1, conj||advmod: 3,
punct||advmod: 6, obj||conj: 2, nummod||conj: 1, obj||obj: 2, nmod:poss||case:
1, punct||compound: 4, nummod||nsubj: 2, compound||advmod: 1, amod||amod: 1,
compound:redup||obj: 2, compound:redup||conj: 5, case||obj: 1, nsubj||advmod: 1,
amod||advmod: 1, dislocated||csubj: 1, flat||nummod: 3, acl||nmod: 1,
advmod||dislocated: 1, advmod||nsubj: 2, advmod||advmod: 2, nmod||nmod:poss: 1,
nmod||amod: 2, flat||nmod: 4, nummod||case: 1, dislocated||acl: 1,
flat||nmod:poss: 3, nsubj||nmod:poss: 2, obl||xcomp: 1, advmod||conj: 3,
nmod||advcl:cond: 1, advcl||nmod:poss: 1, advmod||nmod:poss: 1, cc||advcl: 1,
amod||obj: 1, obj||acl: 1, advmod||ccomp: 2, obl||acl: 1, punct||det: 1,
advmod||case: 1, amod||ccomp: 1, flat||conj: 1, cc||obl: 1, compound||nmod: 2,
punct||discourse: 1, nmod||ccomp: 1, advcl||acl: 1, nmod||cc: 1, advmod||obj: 1,
nummod||advmod: 1, compound||nummod: 1, acl||flat: 1, punct||iobj: 1,
nmod:poss||acl: 1, dislocated||nmod:poss: 1, conj||advcl: 1, nummod||ccomp: 1,
nsubj||advcl:cond: 1, conj||cc:preconj: 1, punct||punct: 1, advmod||acl: 1,
parataxis||advmod: 1, obj||xcomp: 1, punct||xcomp: 1, obj||nmod: 1,
appos||nmod:poss: 1
⚠ The following labels were found only in the train data:
punct||amod, appos||obj, punct||punct, amod||ccomp, case||advcl, punct||appos,
obl||xcomp, nmod||cc, compound||nummod, nummod||obl, obj||xcomp, nmod||ccomp,
parataxis||advmod, advmod||dislocated, dislocated||nmod:poss, amod||amod,
conj||advcl, nummod||nmod:poss, conj||acl, nmod:poss||acl, nmod:poss||obl,
case||amod, obj||conj, advmod||case, amod||obl, advcl||nmod:poss,
amod||nmod:poss, case||obl, amod||conj, compound||nmod, xcomp, obj||advcl,
advmod:emph||conj, nmod||csubj, fixed, acl||conj, acl||nmod, advcl||advcl,
advmod||acl, obj||nmod, obl||ccomp, nmod:poss||case, advmod||nsubj, nmod||mark,
punct||advmod, punct||discourse, advmod||appos, nummod||conj, dislocated||obj,
advmod||nmod:poss, conj||appos, nmod:poss||nmod:poss, nmod||amod, advmod||obj,
obj||iobj, cc||appos, ccomp||advcl, acl||obj, nsubj||advmod, appos||nmod:poss,
nsubj||advcl, nsubj||appos, compound:redup||conj, conj||iobj, appos||nsubj,
advmod:emph||nsubj, conj||cc:preconj, dislocated||nsubj, cc:preconj,
advmod||ccomp, punct||iobj, compound:redup||amod, nmod||advcl:cond,
advmod||advmod, flat||conj, obl||advcl, punct||csubj, cc||nmod:poss,
case||advmod, parataxis||case, nsubj||advcl:cond, case||orphan, nummod||obj,
case||obj, cc||advcl, punct||xcomp, flat||nmod, compound||nmod:poss,
conj||compound, acl||obl, obj||csubj, nmod:poss||nmod, amod||nmod, acl||nsubj,
nsubj||ccomp, orphan||nsubj, obl||conj, dep, obl||nsubj, punct||cc, flat||appos,
nummod||advmod, cc||ccomp, dislocated||acl, appos||obl, advmod:emph||cc,
amod||advmod, det||nsubj, compound||nsubj, dislocated||csubj, nummod||ccomp,
advmod||advcl, aux, nummod||case, nsubj||cop
⚠ The following labels were found only in the dev data:
iobj:caus, amod||acl, nsubj||obl, punct||nummod, case||nummod, iobj||case,
goeswith, cc||cc, advmod:emph||nummod, compound||conj, nmod||obl,
nmod:poss||conj, advmod||discourse, acl||ccomp, case||ccomp, nummod||nmod,
advmod:emph||nmod:poss, parataxis||obj, nsubj||compound
To train a parser, your data should include at least 20 instances of each label.
⚠ Multiple root labels (ROOT, acl, advcl, obl, nsubj, advmod) found in
training data. spaCy's parser uses a single root label ROOT so this distinction
will not be available.
✘ Found 2 projectivized train sentence(s) with cycles

================================== Summary ==================================
✔ 2 checks passed
⚠ 16 warnings
✘ 1 error

[adrianeboyd]

Unfortunately there's not really a built-in method to show the sentences with cycles. I would hack in a print statement here in spacy/cli/debug_data.py to print eg.reference.text so you know which sentences contain cycles:

spaCy/spacy/cli/debug_data.py

Lines 639 to 640 in df541c6

	if nonproj.contains_cycle(aligned_heads):
	data["n_cycles"] += 1

After you've removed those sentences, you should use spacy convert -n 10 or otherwise join the CoNLL-U sentences into longer docs in your training corpus so that the parser also learns how to split sentences.

And that is quite a lot of projectivized labels with few examples, but you'll just have to see how the performance looks like for dev corpus.

[mehmetilker]

@adrianeboyd thank you for your answers. This version called as edited/improved version over original treebank. Will see the performance one I could run training pipeline. I will try.