🦜 Semantic Search on Langchain Github Issues with Weaviate 🔍

🔍 What's Semantic Search?

Semantic search refers to search algorithms that consider the intent and contextual meaning of search phrases when generating results, rather than solely focusing on keyword matching. The goal is to provide more accurate and relevant results by understanding the semantics, or meaning, behind the query.

📋 How does it work?

• Ingesting Github Issues: We use the Langchain Github Loader to connect to the Langchain Repository and fetch the GitHub issues (nearly 2.000), which are then converted to a pandas dataframe and stored in a pickle file. See ./data-pipeline/ingest.py.

• Generate and Index Vector Embeddings with Weaviate: Weaviate generates vector embeddings at the object level (rather than for individual properties), it includes by default properties that use the text data type, in our case we skip the 'url' field (which will be also not filterable and not searchable) and set up the 'text2vec-openai' vectorizer. Given that our use case values fast queries over loading time, we have opted for the HNSW vector index type, which incrementally builds a multi-layer structure consisting from hierarchical set of proximity graphs (layers).

class_obj = {
        "class": "GitHubIssue",
        "description": "This class contains GitHub Issues from the langchain repository.",
        "vectorIndexType": "hnsw",
        "vectorizer": "text2vec-openai",
        "moduleConfig": {
            "text2vec-openai": {
                "model": "ada",
                "modelVersion": "002",
                "type": "text"
            }
        },
        "properties": [
            {
                "name": "title",
                "dataType": ["text"]
            },
            {
                "name": "url",
                "dataType": ["text"],
                "indexFilterable": False,  
                "indexSearchable": False,
                "vectorizePropertyName": False
            },
            {
                "name": "description",
                "dataType": ["text"]
            },
            {
                "name": "creator",
                "dataType": ["text"],
            },
            {
                "name": "created_at",
                "dataType": ["date"]
            },
            {
                "name": "state",
                "dataType": ["text"],
            },
        ]
    }

The ingestion follows in batches of 100 records:

with client.batch as batch: 
    batch.batch_size = 100
    for item in df.itertuples():
        properties = {
            "title": item.title,
            "url": item.url,
            "labels": item.labels,
            "description": item.description,
            "creator": item.creator,
            "created_at": item.created_at,
            "state": item.state,
        }

        batch.add_data_object(
            data_object=properties, 
            class_name="GitHubIssue")

• Searching with Weaviate: Our App supports:

Near-Text-Vector-Search:

@st.cache_data
def query_with_near_text(_w_client: weaviate.Client, query, max_results=10) -> pd.DataFrame:
    """
    Search GitHub Issues in Weaviate with Near Text.
    Weaviate converts the input query into a vector through the inference API (OpenAI) and uses that vector as the basis for a vector search.
    """

    response = (
        _w_client.query
        .get("GitHubIssue", ["title", "url", "labels", "description", "created_at", "state"])
        .with_near_text({"concepts": [query]})
        .with_limit(max_results)
        .do()
    )

    data = response["data"]["Get"]["GitHubIssue"]
    return  pd.DataFrame.from_dict(data, orient='columns')

BM25-Search:

@st.cache_data
def query_with_bm25(_w_client: weaviate.Client, query, max_results=10) -> pd.DataFrame:
    """
    Search GitHub Issues in Weaviate with BM25.
    Keyword (also called a sparse vector search) search that looks for objects that contain the search terms in their properties according to 
    the selected tokenization. The results are scored according to the BM25F function. It is .
    """

    response = (
        _w_client.query
        .get("GitHubIssue", ["title", "url", "labels", "description", "created_at", "state"])
        .with_bm25(query=query)
        .with_limit(max_results)
        .with_additional("score")
        .do()
    )

    data = response["data"]["Get"]["GitHubIssue"]
    return  pd.DataFrame.from_dict(data, orient='columns')

Hybrid-Search:

@st.cache_data
def query_with_hybrid(_w_client: weaviate.Client, query, max_results=10) -> pd.DataFrame:
    """
    Search GitHub Issues in Weaviate with BM25.
    Keyword (also called a sparse vector search) search that looks for objects that contain the search terms in their properties according to 
    the selected tokenization. The results are scored according to the BM25F function. It is .
    """

    response = (
        _w_client.query
        .get("GitHubIssue", ["title", "url", "labels", "description", "created_at", "state"])
        .with_hybrid(query=query)
        .with_limit(max_results)
        .with_additional(["score"])
        .do()
    )

🚀 Quickstart

1. Clone the repository:

git@github.com:dcarpintero/github-semantic-search.git

2. Create and Activate a Virtual Environment:

Windows:

py -m venv .venv
.venv\scripts\activate

macOS/Linux

python3 -m venv .venv
source .venv/bin/activate

3. Install dependencies:

pip install -r requirements.txt

4. Ingest Data

python ./data-pipeline/ingest.py

5. Index Data

python ./data-pipeline/index.py

6. Launch Web Application

streamlit run ./app.py

👩‍💻 Streamlit Web App

Demo Web App deployed to Streamlit Cloud and available at https://gh-semantic-search.streamlit.app/

📚 References

• Langchain Document Loaders - Github
• Weaviate Vector Search
• Weaviate BM25 Search
• Weaviate Hybrid Search
• Weaviate Schema Configuration
• Weaviate - How to efficiently add data objects and cross-references to Weaviate
• Get Started with Streamlit Cloud