Flexible metadata harvesting for ecology using large language models

📦 Repository Overview

This repository contains a tool that harvests metadata from dataset landing pages.

Our methods are described in more detail in our paper, and further data analysis and visualisation functions (used in the paper) are found in a separate repo.

---

Getting started:

Please see the installation instructions below. For quickly getting started, see the data-field-filling.ipynb notebook. Alternatively, for a more detailed tutorial, please see our Colab notebook tutorial and our documentation.

---

🧱 Repo Structure

.
├── name/               # description
├── docs/               # Documentation source and build files
├── examples/           # Tutorial files for getting started
├── imgs/               # Images
├── src/                # Source code
├── tests/              # Unit tests
├── webpages/           # Offline dataset webpages for demos
├── llm_apikey.md       # Guide for setting up LLM API access.
├── pyproject.toml      # Python dependencies
└── README.md           # This file

Installation

Prerequisites

• Python 3.9 or higher
• An API key for either OpenAI or Google Gemini (see API Key Setup below)

Install from Pypi

pip install llm-metadata-harvester

If you want to install the scraper dependency, you can use

pip install "llm-metadata-harvester[scrape]"

Install for Development

If you want to modify the code or contribute to the project:

git clone https://github.com/LTER-LIFE/llm-metadata-harvester.git
cd llm-metadata-harvester
pip install -e .

Additional Setup

After installation, you'll need to install Playwright browsers for web scraping:

playwright install

API Key Setup

This tool requires an API key for a Large Language Model service. You can use either OpenAI or Google Gemini.

Quick Setup:

1. Get your API key following the detailed instructions in llm_apikey.md
2. Set your API key as an environment variable:

# For OpenAI
export OPENAI_API_KEY="your-api-key-here"

# For Gemini
export GEMINI_API_KEY="your-api-key-here"

Alternatively, you can create a .env file in your working directory:

OPENAI_API_KEY=your-api-key-here
# or
GEMINI_API_KEY=your-api-key-here

Verify Installation

To verify the installation worked correctly:

import llm_metadata_harvester
print("Installation successful!")

Usage: Harvesting Metadata from a Dataset Landing Page

After installation and API key setup, you can use the wrapper function metadata_harvest to harvest metadata from a dataset landing page.

Python usage (async)

import asyncio
from llm_metadata_harvester.harvester_operations import metadata_harvest

url = "https://example.com/dataset-page"
model_name = "gpt-4o-mini"  # or a Gemini model name

metadata = asyncio.run(
    metadata_harvest(
        model_name=model_name,
        url=url,
        # Optional parameters:
        # dump_format="json",           # one of: "none" (default), "json", "yaml"
        # allow_retrying=True            # retry to fill missing fields
    )
)

print(metadata)

Notes:

• Set your API key in the environment before running (see API Key Setup). The client reads OPENAI_API_KEY for OpenAI or GEMINI_API_KEY for Gemini.
• model_name must be a valid model for your chosen provider.
• If dump_format is set to "json" or "yaml", the extracted metadata will be written to a file in the current directory.

Example

The following image illustrates the step-by-step procedure and intermediate outputs generated by this tool:

Attribution

Code in the modules harvester_operations.py and utils.py was adapted from LightRAG.