[Feature request]: LocalLLM(ollama) support

[rony432]

What do you need?

this is awesome project, but it needs ollama support.
the OpenAI api is the ease way out.

please add support for local LLMs too.

thank you

[xssdoctor]

It has that support. Ollama and Claude. —model. For a list of models run —listmodels. If you have ollama running on your machine it will list the ollama models you have running.

[rony432]

i have ollama but not on this machine(its remote). i am looking to configure it as custom endpoint.
is there a sample config?

[xssdoctor]

I don’t have a custom endpoint coded in yet. I was looking through the ollama python library and I couldn’t find a way to do it. If anyone knows how to do this I’ll add it

[rony432]

check this out
https://docs.crewai.com/how-to/LLM-Connections/#setting-up-ollama
in crewai user can change the endpoint as global or per agent.
please add support model change for fabric too

side quest. integration as tools for crewai to use fabric

[gcapnias]

I believe ollama support needs some refining:

• You cannot use --listmodels unless you provide an OpenAI key. There are various sources of models; I get that. If there is not a key for a source, skip it, provide a message - there is no key for the group - continue to the next source.
• The model names returned from ollama are not correct; they are missing the last letter of their names. For some reason only llama2 returns the correct name.

Some enhancements:

• I would like to see the default model name reported from fabric if there is one. If not, return there is no default model.
• There is no command to get the version of fabric; Version is needed to report where the bugs are reported.

I used ollama v0.1.28, latest commit of fabric. Ollama has multiple models pulled:

codellama:latest        8fdf8f752f6e    3.8 GB  4 weeks ago
gemma:latest            430ed3535049    5.2 GB  2 weeks ago
llama2:latest           78e26419b446    3.8 GB  4 weeks ago
mistral:latest          61e88e884507    4.1 GB  6 weeks ago
neural-chat:latest      89fa737d3b85    4.1 GB  6 weeks ago
phi:latest              e2fd6321a5fe    1.6 GB  4 weeks ago
qwen:latest             d53d04290064    2.3 GB  4 weeks ago

Fabric reports these models:

Local Models:
codellam
gemm
llama2
mistr
neural-ch
phi
qwen

Looking forward for the next drop,
George J.

[xssdoctor]

fixed. try it now without an api key and let me know if it works

[ghzgod]

Still does not work, additionally fabric does not distinguish between the variants of models (mistral for example). Please see below's output

NAME                         	ID          	SIZE  	MODIFIED
codellama:latest             	8fdf8f752f6e	3.8 GB	3 months ago
deepseek-coder:1.3b-base-q4_1	40fcd5fa2517	856 MB	3 months ago
llama2:latest                	fe938a131f40	3.8 GB	3 months ago
mistral:instruct             	61e88e884507	4.1 GB	29 minutes ago
mistral:latest               	1ab49bc0b6a8	4.1 GB	3 months ago
nomic-embed-text:latest      	0a109f422b47	274 MB	2 weeks ago
(base) josh@M2-Pro:~$ fabric --listmodels
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
GPT Models:

Local Models:
codellama
deepseek-coder
llama2
mistral
mistral
nomic-embed-text

Claude Models:
claude-3-opus-20240229

To fix the output of the model regarding ollama line 298 needs to be modified on utils to say

fullOllamaList.append(model["name"]
instead of
fullOllamaList.append(model["name"].split(":")[0])

[ghzgod]

Ok I think I got it working with ollama models. Please see the below code from my working utils.py

import requests
import os
from openai import OpenAI
import asyncio
import pyperclip
import sys
import platform
from dotenv import load_dotenv
import zipfile
import tempfile
import re
import shutil

current_directory = os.path.dirname(os.path.realpath(__file__))
config_directory = os.path.expanduser("~/.config/fabric")
env_file = os.path.join(config_directory, ".env")


class Standalone:
    def __init__(self, args, pattern="", env_file="~/.config/fabric/.env"):
        """Initialize the class with the provided arguments and environment file.

        Args:
            args: The arguments for initialization.
            pattern: The pattern to be used (default is an empty string).
            env_file: The path to the environment file (default is "~/.config/fabric/.env").

        Returns:
            None

        Raises:
            KeyError: If the "OPENAI_API_KEY" is not found in the environment variables.
            FileNotFoundError: If no API key is found in the environment variables.
        """

        # Expand the tilde to the full path
        env_file = os.path.expanduser(env_file)
        load_dotenv(env_file)
        try:
            apikey = os.environ["OPENAI_API_KEY"]
            self.client = OpenAI()
            self.client.api_key = apikey
        except:
            print("No API key found. Use the --apikey option to set the key")
        self.local = False
        self.config_pattern_directory = config_directory
        self.pattern = pattern
        self.args = args
        self.model = args.model
        self.claude = False
        sorted_gpt_models, ollamaList, claudeList = self.fetch_available_models()
        self.local = self.model.strip() in ollamaList
        self.claude = self.model.strip() in claudeList

    async def localChat(self, messages):
        from ollama import AsyncClient

        response = await AsyncClient().chat(model=self.model, messages=messages)
        print(response["message"]["content"])

    async def localStream(self, messages):
        from ollama import AsyncClient

        async for part in await AsyncClient().chat(
            model=self.model, messages=messages, stream=True
        ):
            print(part["message"]["content"], end="", flush=True)

    async def claudeStream(self, system, user):
        from anthropic import AsyncAnthropic

        self.claudeApiKey = os.environ["CLAUDE_API_KEY"]
        Streamingclient = AsyncAnthropic(api_key=self.claudeApiKey)
        async with Streamingclient.messages.stream(
            max_tokens=4096,
            system=system,
            messages=[user],
            model=self.model,
            temperature=0.0,
            top_p=1.0,
        ) as stream:
            async for text in stream.text_stream:
                print(text, end="", flush=True)
            print()

        message = await stream.get_final_message()

    async def claudeChat(self, system, user):
        from anthropic import Anthropic

        self.claudeApiKey = os.environ["CLAUDE_API_KEY"]
        client = Anthropic(api_key=self.claudeApiKey)
        message = client.messages.create(
            max_tokens=4096,
            system=system,
            messages=[user],
            model=self.model,
            temperature=0.0,
            top_p=1.0,
        )
        print(message.content[0].text)

    def streamMessage(self, input_data: str, context=""):
        """Stream a message and handle exceptions.

        Args:
            input_data (str): The input data for the message.

        Returns:
            None: If the pattern is not found.

        Raises:
            FileNotFoundError: If the pattern file is not found.
        """

        wisdomFilePath = os.path.join(
            config_directory, f"patterns/{self.pattern}/system.md"
        )
        user_message = {"role": "user", "content": f"{input_data}"}
        wisdom_File = os.path.join(current_directory, wisdomFilePath)
        system = ""
        buffer = ""
        if self.pattern:
            try:
                with open(wisdom_File, "r") as f:
                    if context:
                        system = context + "\n\n" + f.read()
                    else:
                        system = f.read()
                    system_message = {"role": "system", "content": system}
                messages = [system_message, user_message]
            except FileNotFoundError:
                print("pattern not found")
                return
        else:
            if context:
                messages = [{"role": "system", "content": context}, user_message]
            else:
                messages = [user_message]
        try:
            if self.local:
                asyncio.run(self.localStream(messages))
            elif self.claude:
                from anthropic import AsyncAnthropic

                asyncio.run(self.claudeStream(system, user_message))
            else:
                stream = self.client.chat.completions.create(
                    model=self.model,
                    messages=messages,
                    temperature=0.0,
                    top_p=1,
                    frequency_penalty=0.1,
                    presence_penalty=0.1,
                    stream=True,
                )
                for chunk in stream:
                    if chunk.choices[0].delta.content is not None:
                        char = chunk.choices[0].delta.content
                        buffer += char
                        if char not in ["\n", " "]:
                            print(char, end="")
                        elif char == " ":
                            print(" ", end="")  # Explicitly handle spaces
                        elif char == "\n":
                            print()  # Handle newlines
                    sys.stdout.flush()
        except Exception as e:
            if "All connection attempts failed" in str(e):
                print(
                    "Error: cannot connect to llama2. If you have not already, please visit https://ollama.com for installation instructions"
                )
            if "CLAUDE_API_KEY" in str(e):
                print(
                    "Error: CLAUDE_API_KEY not found in environment variables. Please run --setup and add the key"
                )
            if "overloaded_error" in str(e):
                print(
                    "Error: Fabric is working fine, but claude is overloaded. Please try again later."
                )
            else:
                print(f"Error: {e}")
                print(e)
        if self.args.copy:
            pyperclip.copy(buffer)
        if self.args.output:
            with open(self.args.output, "w") as f:
                f.write(buffer)

    def sendMessage(self, input_data: str, context=""):
        """Send a message using the input data and generate a response.

        Args:
            input_data (str): The input data to be sent as a message.

        Returns:
            None

        Raises:
            FileNotFoundError: If the specified pattern file is not found.
        """

        wisdomFilePath = os.path.join(
            config_directory, f"patterns/{self.pattern}/system.md"
        )
        user_message = {"role": "user", "content": f"{input_data}"}
        wisdom_File = os.path.join(current_directory, wisdomFilePath)
        system = ""
        if self.pattern:
            try:
                with open(wisdom_File, "r") as f:
                    if context:
                        system = context + "\n\n" + f.read()
                    else:
                        system = f.read()
                    system_message = {"role": "system", "content": system}
                messages = [system_message, user_message]
            except FileNotFoundError:
                print("pattern not found")
                return
        else:
            if context:
                messages = [{"role": "system", "content": context}, user_message]
            else:
                messages = [user_message]
        try:
            if self.local:
                asyncio.run(self.localChat(messages))
            elif self.claude:
                asyncio.run(self.claudeChat(system, user_message))
            else:
                response = self.client.chat.completions.create(
                    model=self.model,
                    messages=messages,
                    temperature=0.0,
                    top_p=1,
                    frequency_penalty=0.1,
                    presence_penalty=0.1,
                )
                print(response.choices[0].message.content)
        except Exception as e:
            if "All connection attempts failed" in str(e):
                print(
                    "Error: cannot connect to llama2. If you have not already, please visit https://ollama.com for installation instructions"
                )
            if "CLAUDE_API_KEY" in str(e):
                print(
                    "Error: CLAUDE_API_KEY not found in environment variables. Please run --setup and add the key"
                )
            if "overloaded_error" in str(e):
                print(
                    "Error: Fabric is working fine, but claude is overloaded. Please try again later."
                )
            if "Attempted to call a sync iterator on an async stream" in str(e):
                print(
                    "Error: There is a problem connecting fabric with your local ollama installation. Please visit https://ollama.com for installation instructions. It is possible that you have chosen the wrong model. Please run fabric --listmodels to see the available models and choose the right one with fabric --model <model> or fabric --changeDefaultModel. If this does not work. Restart your computer (always a good idea) and try again. If you are still having problems, please visit https://ollama.com for installation instructions."
                )
            else:
                print(f"Error: {e}")
                print(e)
        if self.args.copy:
            pyperclip.copy(response.choices[0].message.content)
        if self.args.output:
            with open(self.args.output, "w") as f:
                f.write(response.choices[0].message.content)

    def fetch_available_models(self):
        gptlist = []
        fullOllamaList = []
        claudeList = ["claude-3-opus-20240229"]
        try:
            headers = {"Authorization": f"Bearer {self.client.api_key}"}
            response = requests.get("https://api.openai.com/v1/models", headers=headers)

            if response.status_code == 200:
                models = response.json().get("data", [])
                # Filter only gpt models
                gpt_models = [
                    model for model in models if model.get("id", "").startswith(("gpt"))
                ]
                # Sort the models alphabetically by their ID
                sorted_gpt_models = sorted(gpt_models, key=lambda x: x.get("id"))

                for model in sorted_gpt_models:
                    gptlist.append(model.get("id"))
            else:
                print(f"Failed to fetch models: HTTP {response.status_code}")
                pass
        except:
            print(
                "No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai"
            )

        import ollama

        try:
            default_modelollamaList = ollama.list()["models"]
            for model in default_modelollamaList:
                fullOllamaList.append(model["name"])
        except:
            fullOllamaList = []
        return gptlist, fullOllamaList, claudeList

    def get_cli_input(self):
        """aided by ChatGPT; uses platform library
        accepts either piped input or console input
        from either Windows or Linux

        Args:
            none
        Returns:
            string from either user or pipe
        """
        system = platform.system()
        if system == "Windows":
            if not sys.stdin.isatty():  # Check if input is being piped
                return sys.stdin.read().strip()  # Read piped input
            else:
                # Prompt user for input from console
                return input("Enter Question: ")
        else:
            return sys.stdin.read()


class Update:
    def __init__(self):
        """Initialize the object with default values."""
        self.repo_zip_url = (
            "https://github.com/danielmiessler/fabric/archive/refs/heads/main.zip"
        )
        self.config_directory = os.path.expanduser("~/.config/fabric")
        self.pattern_directory = os.path.join(self.config_directory, "patterns")
        os.makedirs(self.pattern_directory, exist_ok=True)
        print("Updating patterns...")
        self.update_patterns()  # Start the update process immediately

    def update_patterns(self):
        """Update the patterns by downloading the zip from GitHub and extracting it."""
        with tempfile.TemporaryDirectory() as temp_dir:
            zip_path = os.path.join(temp_dir, "repo.zip")
            self.download_zip(self.repo_zip_url, zip_path)
            extracted_folder_path = self.extract_zip(zip_path, temp_dir)
            # The patterns folder will be inside "fabric-main" after extraction
            patterns_source_path = os.path.join(
                extracted_folder_path, "fabric-main", "patterns"
            )
            if os.path.exists(patterns_source_path):
                # If the patterns directory already exists, remove it before copying over the new one
                if os.path.exists(self.pattern_directory):
                    shutil.rmtree(self.pattern_directory)
                shutil.copytree(patterns_source_path, self.pattern_directory)
                print("Patterns updated successfully.")
            else:
                print("Patterns folder not found in the downloaded zip.")

    def download_zip(self, url, save_path):
        """Download the zip file from the specified URL."""
        response = requests.get(url)
        response.raise_for_status()  # Check if the download was successful
        with open(save_path, "wb") as f:
            f.write(response.content)
        print("Downloaded zip file successfully.")

    def extract_zip(self, zip_path, extract_to):
        """Extract the zip file to the specified directory."""
        with zipfile.ZipFile(zip_path, "r") as zip_ref:
            zip_ref.extractall(extract_to)
        print("Extracted zip file successfully.")
        return extract_to  # Return the path to the extracted contents


class Alias:
    def __init__(self):
        self.config_files = []
        home_directory = os.path.expanduser("~")
        self.patterns = os.path.join(home_directory, ".config/fabric/patterns")
        if os.path.exists(
            os.path.join(home_directory, ".config/fabric/fabric-bootstrap.inc")
        ):
            self.config_files.append(
                os.path.join(home_directory, ".config/fabric/fabric-bootstrap.inc")
            )
        self.remove_all_patterns()
        self.add_patterns()
        print("Aliases added successfully. Please restart your terminal to use them.")

    def add(self, name, alias):
        for file in self.config_files:
            with open(file, "a") as f:
                f.write(f"alias {name}='{alias}'\n")

    def remove(self, pattern):
        for file in self.config_files:
            # Read the whole file first
            with open(file, "r") as f:
                wholeFile = f.read()

            # Determine if the line to be removed is in the file
            target_line = f"alias {pattern}='fabric --pattern {pattern}'\n"
            if target_line in wholeFile:
                # If the line exists, replace it with nothing (remove it)
                wholeFile = wholeFile.replace(target_line, "")

                # Write the modified content back to the file
                with open(file, "w") as f:
                    f.write(wholeFile)

    def remove_all_patterns(self):
        allPatterns = os.listdir(self.patterns)
        for pattern in allPatterns:
            self.remove(pattern)

    def find_line(self, name):
        for file in self.config_files:
            with open(file, "r") as f:
                lines = f.readlines()
            for line in lines:
                if line.strip("\n") == f"alias ${name}='{alias}'":
                    return line

    def add_patterns(self):
        allPatterns = os.listdir(self.patterns)
        for pattern in allPatterns:
            self.add(pattern, f"fabric --pattern {pattern}")


class Setup:
    def __init__(self):
        """Initialize the object.

        Raises:
            OSError: If there is an error in creating the pattern directory.
        """

        self.config_directory = os.path.expanduser("~/.config/fabric")
        self.pattern_directory = os.path.join(self.config_directory, "patterns")
        os.makedirs(self.pattern_directory, exist_ok=True)
        self.env_file = os.path.join(self.config_directory, ".env")
        self.gptlist = []
        self.fullOllamaList = []
        self.claudeList = ["claude-3-opus-20240229"]
        load_dotenv(self.env_file)
        try:
            openaiapikey = os.environ["OPENAI_API_KEY"]
            self.openaiapi_key = openaiapikey
        except:
            pass
        try:
            self.fetch_available_models()
        except:
            pass

    def fetch_available_models(self):
        headers = {"Authorization": f"Bearer {self.openaiapi_key}"}

        response = requests.get("https://api.openai.com/v1/models", headers=headers)

        if response.status_code == 200:
            models = response.json().get("data", [])
            # Filter only gpt models
            gpt_models = [
                model for model in models if model.get("id", "").startswith(("gpt"))
            ]
            # Sort the models alphabetically by their ID
            sorted_gpt_models = sorted(gpt_models, key=lambda x: x.get("id"))

            for model in sorted_gpt_models:
                self.gptlist.append(model.get("id"))
        else:
            print(f"Failed to fetch models: HTTP {response.status_code}")
            pass
        import ollama

        try:
            default_modelollamaList = ollama.list()["models"]
            for model in default_modelollamaList:
                self.fullOllamaList.append(model["name"])
        except:
            self.fullOllamaList = []
        allmodels = self.gptlist + self.fullOllamaList + self.claudeList
        return allmodels

    def api_key(self, api_key):
        """Set the OpenAI API key in the environment file.

        Args:
            api_key (str): The API key to be set.

        Returns:
            None

        Raises:
            OSError: If the environment file does not exist or cannot be accessed.
        """
        api_key = api_key.strip()
        if not os.path.exists(self.env_file) and api_key:
            with open(self.env_file, "w") as f:
                f.write(f"OPENAI_API_KEY={api_key}")
            print(f"OpenAI API key set to {api_key}")
        elif api_key:
            # erase the line OPENAI_API_KEY=key and write the new key
            with open(self.env_file, "r") as f:
                lines = f.readlines()
            with open(self.env_file, "w") as f:
                for line in lines:
                    if "OPENAI_API_KEY" not in line:
                        f.write(line)
                f.write(f"OPENAI_API_KEY={api_key}")

    def claude_key(self, claude_key):
        """Set the Claude API key in the environment file.

        Args:
            claude_key (str): The API key to be set.

        Returns:
            None

        Raises:
            OSError: If the environment file does not exist or cannot be accessed.
        """
        claude_key = claude_key.strip()
        if os.path.exists(self.env_file) and claude_key:
            with open(self.env_file, "r") as f:
                lines = f.readlines()
            with open(self.env_file, "w") as f:
                for line in lines:
                    if "CLAUDE_API_KEY" not in line:
                        f.write(line)
                f.write(f"CLAUDE_API_KEY={claude_key}")
        elif claude_key:
            with open(self.env_file, "w") as f:
                f.write(f"CLAUDE_API_KEY={claude_key}")

    def update_fabric_command(self, line, model):
        fabric_command_regex = re.compile(
            r"(alias.*fabric --pattern\s+\S+.*?)( --model.*)?'"
        )
        match = fabric_command_regex.search(line)
        if match:
            base_command = match.group(1)
            # Provide a default value for current_flag
            current_flag = match.group(2) if match.group(2) else ""
            new_flag = ""
            new_flag = f" --model {model}"
            # Update the command if the new flag is different or to remove an existing flag.
            # Ensure to add the closing quote that was part of the original regex
            return f"{base_command}{new_flag}'\n"
        else:
            return line  # Return the line unmodified if no match is found.

    def update_fabric_alias(self, line, model):
        fabric_alias_regex = re.compile(r"(alias fabric='[^']+?)( --model.*)?'")
        match = fabric_alias_regex.search(line)
        if match:
            base_command, current_flag = match.groups()
            new_flag = f" --model {model}"
            # Update the alias if the new flag is different or to remove an existing flag.
            return f"{base_command}{new_flag}'\n"
        else:
            return line  # Return the line unmodified if no match is found.

    def clear_alias(self, line):
        fabric_command_regex = re.compile(r"(alias fabric='[^']+?)( --model.*)?'")
        match = fabric_command_regex.search(line)
        if match:
            base_command = match.group(1)
            return f"{base_command}'\n"
        else:
            return line  # Return the line unmodified if no match is found.

    def clear_env_line(self, line):
        fabric_command_regex = re.compile(
            r"(alias.*fabric --pattern\s+\S+.*?)( --model.*)?'"
        )
        match = fabric_command_regex.search(line)
        if match:
            base_command = match.group(1)
            return f"{base_command}'\n"
        else:
            return line  # Return the line unmodified if no match is found.

    def pattern(self, line):
        fabric_command_regex = re.compile(r"(alias fabric='[^']+?)( --model.*)?'")
        match = fabric_command_regex.search(line)
        if match:
            base_command = match.group(1)
            return f"{base_command}'\n"
        else:
            return line  # Return the line unmodified if no match is found.

    def clean_env(self):
        """Clear the DEFAULT_MODEL from the environment file.

        Returns:
            None
        """
        user_home = os.path.expanduser("~")
        sh_config = None
        # Check for shell configuration files
        if os.path.exists(
            os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")
        ):
            sh_config = os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")
        else:
            print("No environment file found.")
        if sh_config:
            with open(sh_config, "r") as f:
                lines = f.readlines()
            with open(sh_config, "w") as f:
                for line in lines:
                    modified_line = line
                    # Update existing fabric commands
                    if "fabric --pattern" in line:
                        modified_line = self.clear_env_line(modified_line)
                    elif "fabric=" in line:
                        modified_line = self.clear_alias(modified_line)
                    f.write(modified_line)
            self.remove_duplicates(env_file)
        else:
            print("No shell configuration file found.")

    def default_model(self, model):
        """Set the default model in the environment file.

        Args:
            model (str): The model to be set.
        """
        model = model.strip()
        if model:
            # Write or update the DEFAULT_MODEL in env_file
            allModels = self.claudeList + self.fullOllamaList + self.gptlist
            if model not in allModels:
                print(
                    f"Error: {model} is not a valid model. Please run fabric --listmodels to see the available models."
                )
                sys.exit()

        # Compile regular expressions outside of the loop for efficiency

        user_home = os.path.expanduser("~")
        sh_config = None
        # Check for shell configuration files
        if os.path.exists(
            os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")
        ):
            sh_config = os.path.join(user_home, ".config/fabric/fabric-bootstrap.inc")

        if sh_config:
            with open(sh_config, "r") as f:
                lines = f.readlines()
            with open(sh_config, "w") as f:
                for line in lines:
                    modified_line = line
                    # Update existing fabric commands
                    if "fabric --pattern" in line:
                        modified_line = self.update_fabric_command(modified_line, model)
                    elif "fabric=" in line:
                        modified_line = self.update_fabric_alias(modified_line, model)
                    f.write(modified_line)
            print(
                f"""Default model changed to {
                  model}. Please restart your terminal to use it."""
            )
        else:
            print("No shell configuration file found.")

    def remove_duplicates(self, filename):
        unique_lines = set()
        with open(filename, "r") as file:
            lines = file.readlines()

        with open(filename, "w") as file:
            for line in lines:
                if line not in unique_lines:
                    file.write(line)
                    unique_lines.add(line)

    def patterns(self):
        """Method to update patterns and exit the system.

        Returns:
            None
        """

        Update()

    def run(self):
        """Execute the Fabric program.

        This method prompts the user for their OpenAI API key, sets the API key in the Fabric object, and then calls the patterns method.

        Returns:
            None
        """

        print("Welcome to Fabric. Let's get started.")
        apikey = input(
            "Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.\n"
        )
        self.api_key(apikey.strip())
        print(
            "Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.\n"
        )
        claudekey = input()
        self.claude_key(claudekey.strip())
        self.patterns()


class Transcribe:
    def youtube(video_id):
        """
        This method gets the transciption
        of a YouTube video designated with the video_id

        Input:
            the video id specifing a YouTube video
            an example url for a video: https://www.youtube.com/watch?v=vF-MQmVxnCs&t=306s
            the video id is vF-MQmVxnCs&t=306s

        Output:
            a transcript for the video

        Raises:
            an exception and prints error


        """
        try:
            transcript_list = YouTubeTranscriptApi.get_transcript(video_id)
            transcript = ""
            for segment in transcript_list:
                transcript += segment["text"] + " "
            return transcript.strip()
        except Exception as e:
            print("Error:", e)
            return None


class AgentSetup:
    def apiKeys(self):
        """Method to set the API keys in the environment file.

        Returns:
            None
        """

        print("Welcome to Fabric. Let's get started.")
        browserless = input("Please enter your Browserless API key\n")
        serper = input("Please enter your Serper API key\n")

        # Entries to be added
        browserless_entry = f"BROWSERLESS_API_KEY={browserless}"
        serper_entry = f"SERPER_API_KEY={serper}"

        # Check and write to the file
        with open(env_file, "r+") as f:
            content = f.read()

            # Determine if the file ends with a newline
            if content.endswith("\n"):
                # If it ends with a newline, we directly write the new entries
                f.write(f"{browserless_entry}\n{serper_entry}\n")
            else:
                # If it does not end with a newline, add one before the new entries
                f.write(f"\n{browserless_entry}\n{serper_entry}\n")

[xssdoctor]

Thanks! good looking out. I fixed it. Make sure it works for you and let me know

[tgzsolt]

it can use different Ollama location (not just the default localhost and port) if we change
AsyncClient()
to something like this:
AsyncClient(host=os.environ["OLLAMA_HOST"])
In the setup we could ask for OLLAMA_HOST and put in .env (the default could be the default ollama url -> http://localhost:11434)
What do you think?

... i try to integrate Fabric with Slack as a Bot, using local Ollama with local LLM in different docker containers

[xssdoctor]

Well currently I am using the ollama python library, and I can't find a way to change the default ollama url in that library in the documentation. If you can figure that out I'll incorporate it

[tgzsolt]

https://github.com/ollama/ollama-python/blob/main/ollama/_client.py

class AsyncClient(BaseClient):
  def __init__(self, host: Optional[str] = None, **kwargs) -> None:
    super().__init__(httpx.AsyncClient, host, **kwargs)

"host" is an optional paramteter of AsyncClient

[xssdoctor]

OK I added it as --remoteOllamaServer. Please test it and make sure it works and I didn't break anything :)

[hobbytp]

Seems it works. My steps are as below:

##I am not sure if the latest "--model qwen" is needed, seems not.
$ fabric --setup  --remoteOllamaServer http://127.0.0.1:11434 --model qwen
Welcome to Fabric. Let's get started.
Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.
there is no key for the group
OpenAI API key set to there is no key for the group
Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.

there is no key for the group
Please enter your YouTube API key. If you do not have one, or if you have already entered it, press enter.

there is no key for the group
Updating patterns...
Downloaded zip file successfully.
Extracted zip file successfully.
Patterns updated successfully.
Aliases added successfully. Please restart your terminal to use them.

# fabric --listmodels
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
GPT Models:

Local Models:
codellama:latest
gemma:latest
mistral:latest
qwen:latest

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-2.1

cat t.txt | fabric --model qwen:latest --pattern summarize
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai

<--it output the summary info correctly here-->

[rony432]

the update was very fast. thank you

a few improvement notes
there should be a ~/fabric.conf file with contents of end point addresses for ollama endpoints

#--remoteOllamaServer
default="http://localhost:11434" #default path
#extra path's
s1="http://127.0.0.2:11434"
s2="http://127.0.0.3:11434"
s3="http://127.0.0.4:11434"

and so on....

and a System wide env variable for ~/fabric.conf file location, perhaps user needs a different path for it, can setup like this

FABRIC_CONF="/root/hidden-location/fabric.conf"
server names (s1) should be defined by user

fabric automatically look for this file and read it as ollama endpoints

in terminal calling fabric like this
list models from s1 ollama

fabric s1 --listmodels or fabric --s1 --listmodels

if s1 or --s1 is not called it should default to local ollama or GPT4 etc...

and one last thing
fabric --listmodels
should list models from other ollama paths too. :)

i think this is the best way to approach this, what do you guys think

[tgzsolt]

thank you for the fast update

[tgzsolt]

Something not work, what am I missed?
the machine can reach the ollama:
root@5be46746d531:/# curl http://172.17.0.2:11434/api/generate -d '{"model": "mistral", "prompt": "How much is the fish?"}'
...

But i cannot see my local models:
root@5be46746d531:/# fabric --listmodels --remoteOllamaServer http://172.17.0.2:11434
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
Failed to fetch models: HTTP 401
No OpenAI API key found. Please run fabric --setup and add the key if you wish to interact with openai
GPT Models:

Local Models:

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-2.1

I guess instead of this:
...
import ollama
try:
default_modelollamaList = ollama.list()['models']
...
This would be better if host is exist (if --remoteOllamaServer is set):
from ollama import Client
client = Client(host=host)
client.list....

[rony432]

to be openai compatible endpoint on ollama it still needs key try this in your curl request
"OPENAI_API_KEY": "NA"
it should get around error,

question for the dev.
does fabric uses langchain tools openchat for openai api calling or its from openai package?

[xssdoctor]

the openai package

[aicoder2048]

I run Fabric on windows WSL, and run Ollama on native windows. "> fabric --listmodel" don't seem to list any models from ollama.

How do I setup Fabric to access LLM loaded via Ollama in this case? 127.0.0.1 doesn't seem to work, as wsl and navtive windows seem to act as two different system/network.

thanks,
Sean

[rony432]

I run Fabric on windows WSL, and run Ollama on native windows. "> fabric --listmodel" don't seem to list any models from ollama.

How do I setup Fabric to access LLM loaded via Ollama in this case? 127.0.0.1 doesn't seem to work, as wsl and navtive windows seem to act as two different system/network.

thanks, Sean

run ollama on WSL much more stable, windows version is buggy,

[aicoder2048]

I run Fabric on windows WSL, and run Ollama on native windows. "> fabric --listmodel" don't seem to list any models from ollama.
How do I setup Fabric to access LLM loaded via Ollama in this case? 127.0.0.1 doesn't seem to work, as wsl and navtive windows seem to act as two different system/network.
thanks, Sean

run ollama on WSL much more stable, windows version is buggy,

I am sure running both ollama and fabric on WSL would work out. I am also running Open/Ollama WebUI on native windows via docker desktop. I am not sure the dockered-webui would work with WSL-ollama. I will just stay with OAI for fabric now, until the fabric supports native windows.

[chymian]

Since Ollama is OAI compatible and there are tons of OAI-compatible API-services out there, like groq, together, etc.
I suggest, using the defacto industry standard (ATM) OAI library only.
but like in crewai and other, make a portfolio of models and endpoints available.
a lot of AI-apps integrate litellm.ai as library, or one can use it as proxy/router/cache, etc.

as @rony432 has suggested:

a few improvement notes
there should be a ~/fabric.conf file with contents of end point addresses for ollama endpoints

#--remoteOllamaServer
default="http://localhost:11434" #default path
#extra path's
s1="http://127.0.0.2:11434"
s2="http://127.0.0.3:11434"
s3="http://127.0.0.4:11434"

one could predefine models by usecases, i.e.
translate: HOST_1:port_1 model A
math: HOST_1:port_2 model B
coder: HOST_2:port_1 model C
visual: HOST_3:port_1 model D
…

[Papoulos]

Hello,

Firstly, congratulation for this project that seems to be very interessting.

Unfortunatly I can't make it work with Ollama. I have a local Ollama service with Gemma that I can curl with no issue :

curl http://localhost:11434/api/generate -d '{
  "model": "gemma",
  "prompt": "Why is the sky blue?"
}'

{"model":"gemma","created_at":"2024-03-20T09:29:56.609134562Z","response":"The","done":false}
....

But when I try to configure fabric :

   fabric --setup  --remoteOllamaServer http://localhost:11434

It ask me for OpenAI, Claude and YT API but did nothing more :

Welcome to Fabric. Let's get started.
Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.

Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.

Please enter your YouTube API key. If you do not have one, or if you have already entered it, press enter.

Updating patterns...

Downloaded zip file successfully.
Extracted zip file successfully.
Patterns updated successfully.

If I then ask for model list :

fabric --listmodels
Please run --setup to set up your API key and download patterns.

Even with this command :

fabric --listmodels --remoteOllamaServer http://localhost:11434
Please run --setup to set up your API key and download patterns.

In fact every request send me back to the setup :

fabric --list
Please run --setup to set up your API key and download patterns.

There may certainly be something I didn't do right, but I have no clue for the moment.
Thank you

[xssdoctor]

Just run --setup first without the remote server flag. Fabric --setup". If you don't want to use OpenAI or Claude just click enter. Once you do that that message will go away

[Papoulos]

I should have mention that I did that first.
But as it didn't work and I saw in this discussion that it's the command to configure the Ollama server I thought I did worng in the first place :

fabric$ fabric --setup
Welcome to Fabric. Let's get started.
Please enter your OpenAI API key. If you do not have one or if you have already entered it, press enter.

Please enter your claude API key. If you do not have one, or if you have already entered it, press enter.


Please enter your YouTube API key. If you do not have one, or if you have already entered it, press enter.


Updating patterns...
Downloaded zip file successfully.
Extracted zip file successfully.
Patterns updated successfully.



fabric$ fabric --listmodels
Please run --setup to set up your API key and download patterns.


fabric$ curl http://localhost:11434/api/generate -d '{
  "model": "gemma",
  "prompt": "Why is the sky blue?"
}'
{"model":"gemma","created_at":"2024-03-20T10:14:58.21685401Z","response":"The","done":false}
{"model":"gemma","created_at":"2024-03-20T10:14:58.459595578Z","response":" sky","done":false}

[quiet-ranger]

I have exactly the same problem as @Papoulos.

After

fabric --setup

Skipping keys for OpenAI, Claude and YB, I tried the following:

# fabric --remoteOllamaServer http://127.0.0.1:11434 --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914

Local Models:

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

Also:

# cat Boeing-clean.txt | fabric --remoteOllamaServer http://127.0.0.1:11434 --model llama2 -sp write_essay
Error: Error code: 404 - {'error': {'message': 'The model `llama2` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}
Error code: 404 - {'error': {'message': 'The model `llama2` does not exist', 'type': 'invalid_request_error', 'param': None, 'code': 'model_not_found'}}

A curl request from another console works fine:

$ curl http://127.0.0.1:11434/api/generate -d '{ "model": "llama2", "prompt": "Why is the sky blue?"}'
{"model":"llama2","created_at":"2024-03-20T16:29:54.952782624Z","response":"\n","done":false}
{"model":"llama2","created_at":"2024-03-20T16:29:55.306774474Z","response":"The","done":false}

Incidentally, it would be nice to have an option to skip updating patterns when we use fabric --setup. While we try to figure out how to make this work it is just annoying having to wait for these updates that we know won't yield anything we care about.

[Papoulos]

I try the same command as you did but still got the message for configuration :

fabric$ cat ../test.txt | fabric --remoteOllamaServer http://127.0.0.1:11434 --model gemma -sp write_essay
Please run --setup to set up your API key and download patterns.

[ksylvan]

I'm also running both fabric (latest from git commit c3df1e7 ) - and ollama 0.1.29 on Windows 11:

PS C:\Users\kayvan\src\fabric> ollama -v  
ollama version is 0.1.29
PS C:\Users\kayvan\src\fabric> ollama list
NAME                    ID              SIZE    MODIFIED    
llama2:latest           78e26419b446    3.8 GB  3 weeks ago
llava:latest            8dd30f6b0cb1    4.7 GB  2 days ago
mistral:latest          61e88e884507    4.1 GB  3 days ago
mixtral:instruct        7708c059a8bb    26 GB   3 days ago
mixtral:latest          7708c059a8bb    26 GB   3 weeks ago

And it's working correctly:

PS C:\Users\kayvan\src\fabric> Invoke-WebRequest -Uri http://localhost:11434
                                                                                                                        
StatusCode        : 200
StatusDescription : OK
Content           : Ollama is running
RawContent        : HTTP/1.1 200 OK
                    Date: Thu, 21 Mar 2024 22:10:17 GMT
                    Content-Type: text/plain; charset=utf-8
                    Content-Length: 17

                    Ollama is running
Headers           : {[Date, System.String[]], [Content-Type, System.String[]], [Content-Length, System.String[]]}
Images            : {}
InputFields       : {}
Links             : {}
RawContentLength  : 17
RelationLink      : {}

And here is the local model respnding:

PS C:\Users\kayvan\src\fabric> (Invoke-WebRequest -method POST -Body '{"model":"mistral", "prompt":"Why is the sky blue?", "stream": false}' -uri http://localhost:11434/api/generate ).Content | ConvertFrom-json
                                                                                                                        
model                : mistral
created_at           : 3/21/2024 10:10:53 PM
response             :  The color of the sky appears blue due to a process called scattering. When the Sun emits light, it sends out electromagnetic waves in all parts of the spectrum. However, Earth's atmosphere scatters
                       short-wavelength light (blue and violet) more effectively than longer wavelengths (yellow, orange, and red). As a result, when we look up at the sky, we predominantly see the blue light that has been
                       scattered in all directions. Additionally, some of the violet light gets absorbed by the ozone layer in the stratosphere, leaving us with more blue than violet.

                       However, it's important to note that the color of the sky isn't always blue - during sunrise or sunset, for instance, the sky can take on hues of pink, orange, and red due to the presence of more scattered    
                       longer wavelengths.
done                 : True
context              : {733, 16289, 28793, 28705…}
total_duration       : 4894802400
load_duration        : 3076050200
prompt_eval_count    : 15
prompt_eval_duration : 16438000
eval_count           : 187
eval_duration        : 1802114000

However, fabric still does not see the local models:

PS C:\Users\kayvan\src\fabric> fabric --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

[Gerkinfeltser]

I'm really wondering if you guys just aren't missing the .config/fabric/.env file & fabric isn't setup to access Window's user file path as its Mac/Linux based. From what I understant, the .config/fabric dir gets created when you run that setup stuff but as Windows doesn't recognize ~ has a user's home dir it just may not be able to read work correctly.

This is all just a theory though! I did see the author of fabric say Mac/Linux/WSL are the only officially supported distributions atm. I hope I'm wrong though. ;)

[ksylvan]

@Gerkinfeltser if you see in the fabric --listmodels output above, both OpenAI GPT and Claude models are listed (obviously using my API keys).

The .env file is created exactly where you'd expect it.

PS C:\Users\kayvan\.config\fabric> dir

    Directory: C:\Users\kayvan\.config\fabric

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           3/20/2024  3:53 PM                patterns
-a---           3/18/2024  4:56 PM            247 .env
-a---           3/20/2024  3:53 PM           4214 fabric-bootstrap.inc

[Gerkinfeltser]

Ah, kk. Well glad that works then. Sorry for the misdirection.

Edit: I just attempted to run fabric from Windows & I do seem to be able to access both the wsl ollama install & the windows one. I wonder if there's a way to get more debug-ish info concerning ollama & fabric on your systems.

[ksylvan]

@Gerkinfeltser What version of Windows? What version of Python?

Here's what happens in WSL for me:

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ ip route
default via 172.29.112.1 dev eth0
172.29.112.0/20 dev eth0 proto kernel scope link src 172.29.124.43

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ curl http://172.29.112.1:11434; echo ""
Ollama is running

kayvan@SHAKTI:/mnt/c/Users/kayvan/src/fabric$ fabric --listmodels --remoteOllamaServer http://172.29.11.2:11434
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

And in Windows native, basically the same thing, except I don't specify the alternate Ollama server, since it's on localhost:11434

[Gerkinfeltser]

I'm windows 10 & python 3.11.5.

I'm able to access:

• wsl-based ollama from windows-based fabric
• windows ollama from windows
• wsl ollama from wsl-based fabric

I'm unable to access:

• windows ollama from wsl fabric (but this is probably to be expected as wsl to windows seems to be a bit more involved)

FWIW I've never had to specify --remoteOllamaServer http://localhost:11434 (or the like) when running fabric to access ollama. Just running a `fabric --listmodels does the trick.

Let me know if you want me to check anything else.

[ksylvan]

@Gerkinfeltser let ollama bind to all interfaces by setting its server address to 0.0.0.0 and you'll be able to access Windows ollama from WSL by specifying the gateway address, like in my WSL example above. See this discussion: ollama/ollama#703

I'm starting to think this might be a Windows firewall issue. I'll look into it more later this afternoon.

[systerchristian]

Just chiming in to note that I am also seeing the same issue. Ollama locally works great. Remote Ollama doesn't appear to function. During setup it never queries the remote Ollama server. During a pattern run it will ask GPT to use the specified model.

[ksylvan]

@systerchristian I have some fixes. Stay tuned. The crux of the issue appears to be that as mentioned in Ollama docs, I set OLLAMA_HOST in the environment to 0.0.0.0 to bind to all interfaces (including the WSL internal network).

This causes the olllama.list() call to fail. There are a couple of other minor bugs. I'll create a PR.

PS C:\Users\kayvan> fabric --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-1106-vision-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:
llama2:latest

Claude Models:
claude-3-opus-20240229
claude-3-sonnet-20240229
claude-3-haiku-20240307
claude-2.1

And then:

PS C:\Users\kayvan> fabric --pattern ai --model llama2:latest --text 'Why is the sky blue?'
🌌 Understanding the question...

💡 Mental model of input and question:

1. The sky appears blue due to a phenomenon called Rayleigh scattering, where shorter wavelengths of light (like blue and violet) are scattered more than longer wavelengths (like red and orange).
2. This is why we see the sky as blue instead of a uniform gray or white color.
3. The scientific explanation is based on the principles of physics and the properties of light.
4. The question is asking for a simple and straightforward explanation of this natural phenomenon.

[RodriMora]

Doens't Ollama provide an OpenAI compatible API? Wouldn't is just be needed to change the export OPENAI_BASE_URL=http://127.0.0.1:5000/v1/ to something like this?

I'm using textgen webui using this method and it works great.

[erlisdhima]

I'm getting this error

Error: Client error '404 Not Found' for url 'http://127.0.0.1:11434/v1/models'

Does anyone know what I'm doing wrong?

I have these under ~/.config/fabric/.env

OPENAI_API_KEY=ollama
OPENAI_BASE_URL=http://127.0.0.1:11434/v1/

[JaySurplus]

I'm getting this error

Error: Client error '404 Not Found' for url 'http://127.0.0.1:11434/v1/models'

Does anyone know what I'm doing wrong?

I have these under ~/.config/fabric/.env

OPENAI_API_KEY=ollama
OPENAI_BASE_URL=http://127.0.0.1:11434/v1/

Try with "OPENAI_BASE_URL=https://127.0.0.1:11434/v1/"

[erlisdhima]

@JaySurplus it worked! 🎉 🥳
Thanks a lot!