Query Regarding Configuration of Locally Deployed Models in Semantica Extraction Classes

[harshalizode]

Hello Team,

I am currently exploring the Semantica extraction classes and would like to understand how to configure locally deployed models for both Hugging Face and LLM-based extraction methods.

Specifically, I would like guidance on the following points:

• How to configure locally hosted Hugging Face models within the Semantica extraction pipeline.
• How to integrate locally deployed LLM models instead of cloud/API-based models.
• Whether Semantica provides any standard configuration structure or predefined approach for local model registration.
• Any recommended examples or documentation references for setting up local inference with the extraction classes.

I have gone through the available cookbook/examples, but I would appreciate clarification on the correct implementation approach for locally deployed models.

Thank you in advance for your support and guidance.

[KaifAhmad1]

Hello Team,

I am currently exploring the Semantica extraction classes and would like to understand how to configure locally deployed models for both Hugging Face and LLM-based extraction methods.

Specifically, I would like guidance on the following points:

• How to configure locally hosted Hugging Face models within the Semantica extraction pipeline.
• How to integrate locally deployed LLM models instead of cloud/API-based models.
• Whether Semantica provides any standard configuration structure or predefined approach for local model registration.
• Any recommended examples or documentation references for setting up local inference with the extraction classes.

I have gone through the available cookbook/examples, but I would appreciate clarification on the correct implementation approach for locally deployed models.

Thank you in advance for your support and guidance.

Hi @harshalizode! Here's a concise breakdown for each scenario.

---

Installation

# Core library (always required)
pip install -U semantica

# For local HuggingFace NER / relation / triplet models (method="huggingface")
pip install torch transformers

# For local HuggingFace text-generation (provider="huggingface_llm")
pip install torch transformers accelerate

# For Ollama — install the Ollama app from https://ollama.com, then:
ollama serve           # start the server
ollama pull llama3.1   # download a model

---

1. Local HuggingFace NER / Relation / Triplet Models

Pass method="huggingface" and point huggingface_model= at a local directory or a Hub ID (cached after first download):

from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor

text = "Apple Inc. was founded by Steve Jobs in 1976 in Cupertino."

# ── NER ──────────────────────────────────────────────────────────
ner = NERExtractor(
    method="huggingface",
    huggingface_model="dslim/bert-base-NER",   # or r"C:\models\dslim-bert-base-NER"
    device="cpu",                               # "cuda" if GPU available
)
entities = ner.extract(text, aggregation_strategy="max")

# ── Relations (entities arg is required) ─────────────────────────
rel = RelationExtractor(
    method="huggingface",
    huggingface_model="your-org/your-relation-model",
    device="cpu",
)
relations = rel.extract(text, entities)

# ── Triplets (entities/relations are optional) ───────────────────
trip = TripletExtractor(
    method="huggingface",
    huggingface_model="Babelscape/rebel-large",
    device="cpu",
)
triplets = trip.extract(text)

Note: huggingface_model= goes in the constructor. aggregation_strategy= can be passed to extract().

---

2. Local LLM via Ollama (No API Key Needed)

# prerequisites
ollama serve          # keep running
ollama pull llama3.1

from semantica.semantic_extract import NERExtractor, RelationExtractor, TripletExtractor

text = "Apple Inc. was founded by Steve Jobs in 1976."

ner = NERExtractor(
    method="llm",
    provider="ollama",
    llm_model="llama3.1",                  # matches `ollama list` name
    base_url="http://localhost:11434",     # default; change if Ollama runs elsewhere
    temperature=0.0,
    max_tokens=800,
)
entities = ner.extract(text)

rel = RelationExtractor(
    method="llm", provider="ollama", llm_model="llama3.1",
    base_url="http://localhost:11434", temperature=0.0, max_tokens=1200,
)
relations = rel.extract(text, entities)   # entities required

trip = TripletExtractor(
    method="llm", provider="ollama", llm_model="llama3.1",
    base_url="http://localhost:11434", temperature=0.0, max_tokens=1200,
)
triplets = trip.extract(text)

Useful Ollama kwargs

Kwarg	Purpose	Example
base_url	Ollama server URL	"http://192.168.1.5:11434"
llm_model	Model name in Ollama	"mistral", "phi3", "llama3.1:8b"
temperature	Sampling temperature (0 = deterministic)	0.0
max_tokens	Max output tokens	1024
num_ctx	Context window size	4096
repeat_penalty	Repetition penalty	1.1

---

3. Local HuggingFace Text-Generation Model (Instruction-Tuned)

Use provider="huggingface_llm" to run a causal / instruction-tuned model through the transformers pipeline:

from semantica.semantic_extract import NERExtractor

ner = NERExtractor(
    method="llm",
    provider="huggingface_llm",
    llm_model="microsoft/Phi-3-mini-4k-instruct",  # Hub ID or local path
    device="cpu",   # "cuda" for GPU
    max_tokens=512,
    temperature=0.0,
)
entities = ner.extract(text)

Requires pip install torch transformers accelerate. Works best with instruction-tuned models — base models produce unreliable structured output.

---

4. OpenAI-Compatible Local Server (vLLM / LM Studio)

If your local server exposes an OpenAI-compatible API, point the built-in openai provider at it:

from semantica.semantic_extract import NERExtractor

# vLLM:      vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000
# LM Studio: start server from UI (usually :1234)

ner = NERExtractor(
    method="llm",
    provider="openai",
    llm_model="meta-llama/Llama-3.1-8B-Instruct",  # name the server expects
    base_url="http://localhost:8000/v1",             # your server's base URL
    api_key="EMPTY",                                 # most local servers accept any value
    temperature=0.0,
    max_tokens=800,
)
entities = ner.extract(text)

---

5. Registering a Fully Custom Provider

Subclass BaseProvider and register it once — then use it like any built-in:

import json
import requests
from semantica.semantic_extract.providers import BaseProvider
from semantica.semantic_extract.registry import provider_registry


class MyLocalProvider(BaseProvider):
    def __init__(self, base_url="http://localhost:8080", model="", **kwargs):
        super().__init__(**kwargs)
        self.base_url = base_url
        self.model = model

    def is_available(self):
        try:
            requests.get(self.base_url, timeout=2)
            return True
        except Exception:
            return False

    def generate(self, prompt, **kwargs):
        resp = requests.post(
            f"{self.base_url}/completion",
            json={"prompt": prompt, "model": self.model, **kwargs},
            timeout=60,
        )
        resp.raise_for_status()
        return resp.json().get("content", "")

    def generate_structured(self, prompt, **kwargs):
        try:
            return json.loads(self.generate(prompt, **kwargs))
        except json.JSONDecodeError:
            return {}


# Register once; the name is what you pass as provider= in the extractor
provider_registry.register("my_local", MyLocalProvider)

from semantica.semantic_extract import NERExtractor

ner = NERExtractor(method="llm", provider="my_local", base_url="http://localhost:8080")
entities = ner.extract("Apple was founded by Steve Jobs.")

---

6. Resilience: Fallback Chains

Pass a list to method= — Semantica tries each in order if the previous one fails:

from semantica.semantic_extract import NERExtractor

ner = NERExtractor(
    method=["llm", "ml", "pattern"],   # try Ollama → spaCy → pattern matching
    provider="ollama",
    llm_model="llama3.1",
    base_url="http://localhost:11434",
)
entities = ner.extract(text)

---

Quick Reference

Runtime	provider=	Extra constructor args
HF NER / RE / Triplet model	(use method="huggingface")	huggingface_model=, device=
Ollama	"ollama"	llm_model=, base_url=
HF text-generation	"huggingface_llm"	llm_model=, device=
vLLM / LM Studio	"openai"	llm_model=, base_url=, api_key="EMPTY"
Custom runtime	"your_name" after provider_registry.register(…)	whatever your class needs

Heads-up:

• RelationExtractor.extract(text, entities) — entities is always required.
• TripletExtractor.extract(text, entities=None, relations=None) — both are optional.

---

Hope this helps! Let us know which runtime you're using if you run into anything specific.

[harshalizode]

Hello @KaifAhmad1,

Thank you for the detailed response and for sharing multiple examples demonstrating the different configuration possibilities.
The examples helped clarify the approach for configuring locally deployed Hugging Face and LLM models within the Semantica extraction classes. I appreciate the effort taken to cover the various integration scenarios and implementation patterns.
I will go through the shared examples in detail.

Thanks again for your support and guidance.