Add fields metadata suggestions - #99
Conversation
93cfa56 to
608ecf8
Compare
slint
left a comment
slint
left a comment
There was a problem hiding this comment.
LGTM! Some minor things, plus an idea to shelve for later when we do evals.
| ), | ||
| examples=[["mit", "apache-2.0", "cc-by-4.0", "gpl-3.0-only"]], | ||
| ) | ||
| funding: list[str] = Field( |
There was a problem hiding this comment.
| funding: list[str] = Field( | |
| funding_titles: list[str] = Field( |
minor: tbh this whole "parallel lists" is so that gtp-oss-20b manages to produce something... I feel that once we change models we can switch to a nested schema (I would still translate "manually" the results to MetadataSuggestion).
@juliehinge another idea to play with later, when we make it easy to run evals for different Orcha "code variants", is to see if it makes sense to spawn separate agents to extract different parts of the metadata (e.g. one for title + description, one for authors, one for license + copyright, etc.). The idea is to keep the context "lean", have each agent focus on a specific aspect, and then merge the results (either programmatically or with a "review" agent).
608ecf8 to
aad6aaf
Compare
Move the LLM agent construction logic from `activities._llm` to a dedicated module `agent`.
Add fields for metadata sggestions: copyright, funding and license.
aad6aaf to
2d873b2
Compare
Add a `resolve_metadata_suggestions` activity that turns the LLM's extraction into resolved entries. The activity runs after `ExtractMetadata`. Licenses resolve via `/api/vocabularies/licenses/<id>`, falling back to the bare id. Awards resolve by grant number first, then by title (if there is a single hit or the top hit if the acronym is in the extracted award title). If the award cannot be resolved, its title and/or number is passed as a custom award. Add `INVENIO_BASE_URL` to name the instance to resolve against.
`metadata_suggestions.py` held both the flat schema the LLM fills and the typed payload the API returns. Resolved fields have been added, so this commit separates the schema into: - `extracted_metadata.py`: schema passed to the LLM - `resolved_fields.py` - `metadata_suggestions.py`: suggestions envelope built from extracted metadata and resolved fields `extract_metadata_with_llm` now returns `ExtractedMetadata`, the payload is built in the resolve metadata activity.
The host allowlist was private to the PDF download activity, but since the resolve metadata activity also calls an Invenio instance, we move `_http_verify_for_pdf_url` to `app.activities.utils.http_verify`, generalize its error message, and use it in both activities. This commit also renames `PDF_HTTP_ALLOWLIST` to `HTTP_ALLOWLIST` to be more accurate, and exposes `INVENIO_BASE_URL` through the chart.
Set `output_retries=2` on the agent to solve output validation issues, such as the model wrapping the JSON or dropping arequired field. This re-prompts the model with the validation error instead of immediately failing the activity.
2d873b2 to
f7ce162
Compare
ciurana-life
left a comment
ciurana-life
left a comment
There was a problem hiding this comment.
I'd recommend not to rely on wildcard except that wrap entire functions, is not very pythonic and is a code pattern that if needed could be a decorator or a function call (DRY).
|
|
||
| BASE_URL = "https://invenio.test" | ||
|
|
||
| LICENSES = { |
There was a problem hiding this comment.
Recomendation:
https://vcrpy.readthedocs.io/en/latest/
This library records all the http interactions and replays them, so you don't need to write all the responses, you run the tests once and it saves the respones.
There was a problem hiding this comment.
It looks nice. We do the same kind of API mocking in many modules, so maybe it would make sense to have a proof of concept PR and show it to the InvenioRDM community for feedback?
4 participants
Description
feat(schema): add metadata fields
Add fields for metadata sggestions: copyright, funding and license.
refactor(agent): move shared LLM agent builder
Move the LLM agent construction logic from
activities._llmto a dedicated moduleagent.Checklist
Ticks in all boxes and 🟢 on all GitHub actions status checks are required to merge:
Reminder
By using GitHub, you have already agreed to the GitHub’s Terms of Service including that: