Repository navigation
ReDefiNER 1.0.4
A generalist 0.8B model for entity extraction and span labeling, guided by the rulebook you write.
pip install redefinerWeights: jungroo-ai/redefiner-0.8b · Demo: jungroo-ai/redefiner-demo · Benchmarks
Fixes from an end-to-end test of 1.0.3. Same weights; every rulebook with one section per type gives exactly the same output (all 490 rulebooks in the repo encode identically).
Rulebooks
- A type defined twice (two
[TYPE] PERSONsections, or a dict that repeats a class name) is now aValueErrornaming the type and its lines. Before, it was accepted silently, and when another type sat between the two sections their labels were swapped: with PERSON, ORG, PERSON, "Tim Cook" came out as ORG at 0.99 confidence. This applied toextractandlabel. - A dict's
"closing"rules are written under a second[RULES]line, as the docs, the demo anddocs/templates/blank.txtalready did. On CrossNER AI, closing rules scored 75.98 F1 with the header and 75.64 without. No rulebook in the repo has closing rules, so none changes.
Device
tagger.devicesays where the model runs (cpu,cuda:0), andrepr(tagger)shows it.- With the default device, a
RuntimeWarningwhen the model falls back to CPU on a machine with an NVIDIA driver: a CPU-only PyTorch build, or a CUDA build the driver can't run. Passdevice="cpu"to silence it. redefiner serveprints the device at startup, andGET /infoincludes it.
Docs
- README: a "Rule lines" section. One rule per line (wrapping long rules onto continuation lines lowered CrossNER AI F1 by 0.8 and 0.5); rule IDs such as
G1orPE1are optional (removing them changed F1 by −0.02 and +0.6); a rule belongs to the section it sits in; quoted strings under a type are read as examples of it. Type names are case-sensitive. docs/writing-rules.mdquotes its example rules as the single lines they are in the benchmark rulebooks.llms.txt: a short summary of the API and rulebook format for AI assistants and coding agents.