Evaluation result paper#104
Evaluation result paper#104tekrajchhetri wants to merge 50 commits intoperformance_cost_optimizationfrom
Conversation
…nse into evaluation_result_paper
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the documentation, configuration files, and evaluation results for the StructSense NER pipeline. The changes include adding an evaluation section to the README, updating configuration files for various models (Gemini, GPT, Qwen), and refining the output structure for evaluation results. The review comments regarding missing trailing newlines in various files are valid and should be addressed to ensure POSIX compliance.
| @@ -0,0 +1 @@ | |||
| {} No newline at end of file | |||
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
| {} | |
| {} | |
| @@ -0,0 +1,3 @@ | |||
| { | |||
| "judge_method": "auto_approved" | |||
| } No newline at end of file | |||
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
"judge_method": "auto_approved"
}
| config: | ||
| api_base: http://localhost:11434 | ||
| model: nomic-embed-text | ||
|
|
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
model: nomic-embed-text
| config: | ||
| api_base: http://localhost:11434 | ||
| model: nomic-embed-text | ||
|
|
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
model: nomic-embed-text
| config: | ||
| api_base: http://localhost:11434 | ||
| model: nomic-embed-text | ||
|
|
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
model: nomic-embed-text
| config: | ||
| api_base: http://localhost:11434 | ||
| model: nomic-embed-text | ||
|
|
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
model: nomic-embed-text
| config: | ||
| api_base: http://localhost:11434 | ||
| model: nomic-embed-text | ||
|
|
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
model: nomic-embed-text
| --config config_name.yaml \ | ||
| --source input_pdf.pdf \ | ||
| --api_key sk-<your_api_key> | ||
| ``` No newline at end of file |
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
| ``` | |
| --api_key sk-<your_api_key> |
| config: | ||
| api_base: http://localhost:11434 | ||
| model: nomic-embed-text:v1.5 | ||
| model: nomic-embed-text |
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
model: nomic-embed-text
| config: | ||
| api_base: http://localhost:11434 | ||
| model: nomic-embed-text:v1.5 | ||
| model: nomic-embed-text |
There was a problem hiding this comment.
The file is missing a newline character at the end. It's good practice to include a trailing newline for POSIX compliance and better compatibility with various tools.
model: nomic-embed-text
2 participants
This PR contains the following + one minor fix for resource merge.
Benchmark
This directory contains benchmark datasets based on standard evaluation sets.
It includes NCBI and S800, where the
.jsonlfiles serve as the ground-truth annotations.Link:
https://github.com/sensein/structsense/tree/evaluation_result_paper/evaluation/benchmark
NER
This directory contains evaluation results for three papers related to Named Entity Recognition (NER).
Link:
https://github.com/sensein/structsense/tree/evaluation_result_paper/evaluation/ner
Resource Extraction
This directory contains evaluation results for two papers focused on resource extraction.
Link:
https://github.com/sensein/structsense/tree/evaluation_result_paper/evaluation/resource_extraction
Pdf2Reproschema
This directory contains evaluation outputs for the Pdf2Reproschema pipeline.
Link:
https://github.com/sensein/structsense/tree/evaluation_result_paper/evaluation/pdf2reproschema