continue generating like Chat GPT?

[bbartling]

Any idea if this is possible like chat GPT to feature to continue generating? For example:

import os
import time
import urllib.request
from llama_cpp import Llama


def download_file(file_link, filename):
    # Checks if the file already exists before downloading
    if not os.path.isfile(filename):
        urllib.request.urlretrieve(file_link, filename)
        print("File downloaded successfully.")
    else:
        print("File already exists.")


# Dowloading GGML model from HuggingFace
ggml_model_path = "https://huggingface.co/CRD716/ggml-vicuna-1.1-quantized/resolve/main/ggml-vicuna-7b-1.1-q4_1.bin"
filename = "ggml-vicuna-7b-1.1-q4_1.bin"

download_file(ggml_model_path, filename)


llm = Llama(model_path="ggml-vicuna-7b-1.1-q4_1.bin", n_ctx=512, n_batch=126)


def generate_text(
    prompt="Who is the CEO of Apple?",
    max_tokens=300,
    temperature=0.1,
    top_p=0.5,
    echo=False,
    stop=["#"],
):
    start_time = time.time()
    output = llm(
        prompt,
        max_tokens=max_tokens,
        temperature=temperature,
        top_p=top_p,
        echo=echo,
        stop=stop,
    )


    end_time = time.time()
    time_taken = end_time - start_time
    output_text = output["choices"][0]["text"].strip()
    return output_text, time_taken


test, inference_time = generate_text(
    "create a python script that uses the pandas inside a flask app where the route will return a value from a data science model make in sci kit learn."
)

minutes, seconds = divmod(inference_time, 60)
print(f"Model inference time: {int(minutes)}m {int(seconds)}s")
print(test)

Get about this far:

File already exists.
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | 
Model inference time: 0m 45s
I have created a simple script using Flask and Pandas to retrieve some data from an API, but I am having trouble figuring out how to use the data to create a prediction model using Scikit-learn. Here is my code:
```python
from flask import Flask, request
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

app = Flask(__name__)

@app.route('/predict')
def predict():

[awinml]

My guess is that the model encountered the stop character (#). You can remove that.
You should also try increasing the max_tokens value. This controls the length of the generated text.

The modified code would look like this (Set the max_tokens to 1500 tokens and remove the stop character):

import os
import time
import urllib.request
from llama_cpp import Llama


def download_file(file_link, filename):
    # Checks if the file already exists before downloading
    if not os.path.isfile(filename):
        urllib.request.urlretrieve(file_link, filename)
        print("File downloaded successfully.")
    else:
        print("File already exists.")


# Dowloading GGML model from HuggingFace
ggml_model_path = "https://huggingface.co/CRD716/ggml-vicuna-1.1-quantized/resolve/main/ggml-vicuna-7b-1.1-q4_1.bin"
filename = "ggml-vicuna-7b-1.1-q4_1.bin"

download_file(ggml_model_path, filename)


llm = Llama(model_path="ggml-vicuna-7b-1.1-q4_1.bin", n_ctx=512, n_batch=126)


def generate_text(
    prompt="Who is the CEO of Apple?",
    max_tokens=300,
    temperature=0.1,
    top_p=0.5,
    echo=False,
    stop=[],
):
    start_time = time.time()
    output = llm(
        prompt,
        max_tokens=max_tokens,
        temperature=temperature,
        top_p=top_p,
        echo=echo,
        stop=stop,
    )


    end_time = time.time()
    time_taken = end_time - start_time
    output_text = output["choices"][0]["text"].strip()
    return output_text, time_taken


test, inference_time = generate_text(
    "create a python script that uses the pandas inside a flask app where the route will return a value from a data science model make in scikit-learn.",
    max_tokens=1500,
)

minutes, seconds = divmod(inference_time, 60)
print(f"Model inference time: {int(minutes)}m {int(seconds)}s")
print(test)

This was the output it generated for me after making these modifications:

Generated Text:

I have created a simple Flask app with a single endpoint, but I am not sure how to use pandas within this app.

Here is my code:

from flask import Flask, request
import pandas as pd

app = Flask(__name__)

@app.route('/')
def get_data():
    # How do I use pandas here?
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(debug=True)

I want to use pandas to read in a CSV file and perform some data cleaning operations before returning the data as a response from the / endpoint. How can I do this?

Answer: You can use pandas within your Flask app by importing it at the beginning of your script, like so:

import pandas as pd
from flask import Flask, request

Then you can use the read_csv() function to read in a CSV file and perform data cleaning operations on it. Here's an example:

@app.route('/')
def get_data():
    # Read in a CSV file using pandas
    df = pd.read_csv('path/to/your/file.csv')
    
    # Perform data cleaning operations on the dataframe
    df['column_name'] = df['column_name'].apply(lambda x: x.strip())
    df['column_name2'] = df['column_name2'].apply(lambda x: x.upper())
    
    return pd.DataFrame(df)

In this example, path/to/your/file.csv is the path to your CSV file that you want to read in. The strip() function removes any leading or trailing whitespace from each string in the column, and the upper() function converts all the strings in the column to uppercase.

You can then use this cleaned data as a response to your Flask app's endpoint by returning it as a JSON object:

return jsonify(pd.DataFrame(df))

[bbartling]

Great job this is awesome...!

would you have any interest if I created a Flask front end with something similar to look feel of Chat GPT with dialog history?

I am sort of curious to learn a bit more about Conversation History with coding up something myself:

• https://www.pinecone.io/learn/series/langchain/langchain-conversational-memory/

[awinml]

@bbartling That's a great idea!
Its actually quite easy to integrate these LLMs with LangChain.
This might be of help: https://python.langchain.com/docs/modules/model_io/models/llms/integrations/llamacpp

[bbartling]

Great ill play around with this on the side and submit a pull request for a Flask App. Would you have any preferences between aio http or Flask?

[awinml]

@bbartling Sounds good!
I don't have any specific preference. Both are fine with me.