Merged
Conversation
semindan
added a commit
that referenced
this pull request
Jan 25, 2026
* Feat/setup (#1) * add multigec setup * modify baseline, because I don't have access to llama, I'm banned * update dependencies * delete redundant files * add initial setup with a couple of methods * Feat/experiments (#2) * add base method * ignore data * add script for generating data * update dependencies * delete init script * add dataset classes * add script for running experiments * add methods * add f1 with overlaps metric * Feat/experiments (#4) * add linting * add pre-commit config * add linting actions * add soft version of f1 with overlaps * change return type to Any * update deps * temporarily turn off mypy * refactor with model_input * update methods * delete registry file * add prompts for each method/dataset * add prompt utils for assembling prompts * remove correction notes from multigec prompts * add script for running a single experiment * put code into functions * add scripts for eval and export * add support for examples in prompts * increase number of tokens to 8192 * add examples to prompts * update dependencies * add script for running an experiment * add function for checking whether spans overlap * add async methods * update deps * fix prompts * ignore results and env * ignore torch cache * rename create_test_data * add allowed labels * refactor methods * delete old files * add zero-length matching * add extra note to prompt builder * add eval script * add config * add constrained processor * update dependencies * ignore logs and results * add constrained method implementation * update prompts * add prompt note about enriched indices * update labels * add data conversion script * enhance note extra insertion * add run scripts * add wmt old labels fallback
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several major improvements to the span labeling codebase, focusing on extensibility, configuration, dataset handling, error analysis, and development tooling. The most significant changes are the introduction of registry-based extensibility for methods and datasets, the addition of a robust configuration system using Pydantic, enhanced dataset preprocessing, a new error analysis utility, and the setup of linting and pre-commit hooks for code quality.
Core extensibility and architecture:
MethodRegistryMetaandDatasetRegistryMetametaclasses inspan_labeling/base.pyto automatically register span labeling methods and datasets, enabling easy extensibility and dynamic lookup. AddedMethodRegistryandDatasetRegistryaccess classes for retrieval and listing.span_labeling/dataset.pyto inherit fromDatasetBase(now using registry metaclass), standardized dataset attributes, and improved preprocessing logic for each dataset type. [1] [2]Configuration and settings:
span_labeling/config.pywith a new configuration system based on Pydantic, supporting environment variables and YAML config loading for model and experiment settings.Error analysis and utilities:
span_labeling/error_analysis.py, a utility for analyzing prediction errors and categorizing them (e.g., empty response, invalid labels, truncation, schema errors), including sample usage for batch analysis.Development tooling and code quality:
.github/workflows/lint.yamlfor GitHub Actions-based linting (ruff, mypy) and.pre-commit-config.yamlfor pre-commit hooks to enforce code formatting and static analysis. [1] [2]pyproject.tomlto include new dependencies for span labeling, configuration, and development tooling, and defined adevdependency group.Other improvements:
span_labeling/methods/__init__.pyto explicitly import and expose available span labeling method modules.