LLM for engineering questions using langchain4j (optionally on Azure)

Vision

• Questions about code (preferrably our company repositories) can be answered

  • similar to co-pilot

  • more specific to our company

  • eg 'how to search for a partner?', how to use the partner service, …​

Monday

• LangChain4J learnings

  • RAG (Retrieval Augmented Generation) Process

  • Step 1: ("setup time") create embedded Model of own documents (e.g. Java files)

  • Step 2: ("runtime") ask send question tor embeddings. → 1-n results

  • Step 3: ("runtime") send result(s) and question to LLM

we used OpenAI

Tuesday

• Ollama (local LLM instead of OpenAI)

  • ollama in local docker container (with LLMs: 'codegemmaa', codellama'), and natively on Mac

  • used ollama-ui as a chat-gui (very similar to chatGPT UI) against ollama server

• Inspect copilot protocol (which requests are sent from VS-Code to github) using an nginx proxy with logging

• Redirect copilot requests to local ollama server via ollama-copilat (go program)

Wednesday

• Tried to create embeddings for local Java files → results not very promising

  • → we need a language (Java) specific splitter. Not available in langchain4J but in langchain-ai (Python)

Thursday

index Java codebase

• langchain-ai with openai (pay) → works quite fast, good results

• running locally, using all-MiniLM-L6-v2:

  • generate embeddings with python → chromadb

  • query with Java (langchain4j) using embeddings from chromadb against ollama-LLM → works, but slow, results OK, GPU improves performance a lot

ToDo

• ci/cd pipeline for such an application

• how can we offer the service sustainably

• learning, learning

• TDD such an app

• check out sourcegraph.com

Done

• Checked out Ollama https://docs.langchain4j.dev/integrations/language-models/ollama

• explored the langchain4j example repository

• found the InMemoryEmbeddingStoreExample which

  • load files/text into the model ("I like football", …​)

  • and ask questions "which sport do you like" → "football"

• watched very good presentation on an RAG use case: https://youtu.be/J-3n7xs98Kc?si=xiSmOOrVC3IcbxEP

  • 25:05: interface EmbeddingModel: AllMiniLmL6V2EmbeddingModel vs. OpenAIModel

  • 25:56: EmbeddingStore

  • 27:50: Retriever

  • 28:15: content ingestion explained

  • 28:37: workaround CommandlineRunner

  • 29:45: load document

  • 32:35: "this is RAG", he used it to build the vaadin docs assistant

  • 34:30: tools

  • github:

    • https://github.com/marcushellberg/java-ai-playground

    • https://github.com/marcushellberg/java-chat-with-documents

• see also 3blue1brown series: https://www.youtube.com/watch?v=wjZofJX0v4M&t=2s

• RAG for code: https://cloud.google.com/blog/products/ai-machine-learning/context-aware-code-generation-rag-and-vertex-ai-codey-apis?hl=en

Some findings

Corelia codebase, openai embeddings

Q: "what’s the max length of a street?"

gpt-4 answer: "The maximum length of a street is defined by the constant STREET_MAX_SIZE, which is 35"

codellama:code answer:

  @Size(max = COMPANY_NAME_MAX_SIZE)
  @JsonPropertyDescription("Name of the contact (first and last name). Required when partnerNr not set!")
  private String contactName;

... much more gibberish

codellama answer:

package ch.baloise.corellia.api.entities;

import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonPropertyDescription;
import com.fasterxml.jackson.annotation.JsonValue;

public enum LegalForm {
  PRIVATPERSON("01"),
  EINZELUNTERNEHMEN("10"),
...

llama3 answer: Based on the information you provided, it seems that there is no specific max length mentioned for a street in the code. However, in the SizeConstraint class, there is a constant named STREET_MAX_SIZE which has a value of 35. This suggests that the maximum length of a street in this context is 35 characters.

Conclusion: the main purpose of codellama is is for code completion

Hexagonal architecture codebase with openAI

repo: https://github.com/dziadeusz/hexagonal-architecture-by-example

Question: how can I publish an article on twitter?

Answer:

In order to publish an article on Twitter using the provided code, follow these steps:

1. Create an instance of your Article object that you want to publish. This requires the article’s author and title.

2. Create an instance of the TwitterArticlePublisher class. This class requires an instance of the TwitterClient class.

3. Call the publish method of the TwitterArticlePublisher instance, passing your Article object as an argument.

4. The publish method will create a new ArticleTwitterModel object using the static of method, which formats the tweet with the article’s title and the author’s Twitter ID.

5. The publish method will then call the tweet method of the TwitterClient instance, passing the ArticleTwitterModel object as an argument.

Note: The actual implementation for tweeting via Twitter’s API should be in the tweet method of the TwitterClient. This implementation is not provided in the code.

Here is an example:

java Article article = new Article("AuthorName", "ArticleTitle"); TwitterClient twitterClient = new TwitterClient(); TwitterArticlePublisher twitterPublisher = new TwitterArticlePublisher(twitterClient); twitterPublisher.publish(article);

This will create a tweet with the following format: "Check out the new article >>ArticleTitle<< by AuthorName". done

Hexagonal architecture codebase, local

embeddedModel: chromadb: all-MiniLM-L6-v2 LLM-Model: ollama: codegemma

Question: how can I publish an article on twitter?

Answer:

Step 1: Configure the TwitterClient

• Implement the tweet() method to interact with the Twitter API.

• Ensure that you have the necessary credentials and API keys for Twitter integration.

Step 2: Create a TwitterArticlePublisher

• Inject the TwitterClient instance into the constructor.

• Implement the publish() method to create an ArticleTwitterModel object and call the tweet() method of the TwitterClient.

Step 3: Use the ArticlePublisher

• In your article creation logic, create an ArticleBuilder with the necessary information and call the build() method.

• Inject the ArticlePublisher and call the publishCreationOf() method to publish the article on Twitter.

Code Example:

// In your ArticlePublisher class
private final TwitterArticlePublisher twitterArticlePublisher;

public ArticlePublisher(TwitterArticlePublisher twitterArticlePublisher) {
    this.twitterArticlePublisher = twitterArticlePublisher;
}

public void publishCreationOf(Article article) {
    super.publishCreationOf(article);
    twitterArticlePublisher.publish(article);
}

Additional Notes:

• Ensure that the Twitter account associated with the application has access to the necessary permissions for posting tweets.

• Test the Twitter integration thoroughly before deploying it in production.

• Consider using a social media management platform or API client library for easier integration. done

Links

• Talk to Your Code | Github Repo | Learn How GitHub Co-Pilot & Others Transform Coding

• CODE-LLAMA For Talking to Code Base and Documentation