Ingesting large textfiles through Goals

[MrLindeman]

Hi

I've been struggling to get Auto-Gpt to read large files. Tried everything from using data_ingestion.py, uploading PDF to my own server and converting the PDF into a HTML text page and also having the ai read that page. All of them to no avail. I simply can't get data_ingestion.py to connect to Redis, no matter what I do.

BUT!

I found a way to make this work with a "simple" workaround.

1. Converted PDF into a textfile (UTF-8), using an online service
2. Used Gsplit (free software), to split the textfile into 4KB chunks
3. Copied the chunks into a subdirectory in the workspace for Auto-GPT
4. Using a python script to add a new line in each file "Read (filename{num}.txt) next"
5. Ran Auto-GPT and in goal 1 I set: /subdir/"Read (filename{num}.txt. Store all information in long-term memory before proceeding. At the end of the textfile you will find instructions on what to do next until you reach "filename{num}.txt."

This results in Auto-GPT reading one file after another, it may error out as it goes sometimes (Mine did after the first 92 files, but editing Goal 1 to pick up where it exited will set it off again.

The pythonscript I used looks like this:

import os

# Store the information in long-term memory
file_info = []

# Loop through the file numbers from 1 to n
for num in range(1, <last number in series>):
    current_filename = f"<filename>{num}.txt"
    next_filename = f"<filename>{num+1}.txt"
    line = f"\nRead {next_filename} next"
    file_info.append((current_filename, line))

    # Append the line to the file
    with open(current_filename, "a") as file:
        file.write("\n" + line)

# Store the file information in long-term memory for later use
long_term_memory = file_info

Just change to your own filename without the number.

This has been working great for me. Yes it takes a lot of time for larger files.. I successfully ingested a file with 2.500.000 characters, 404 textfiles.

Good luck out there :)