Add an option to set the chunk size using the

configoration - BROWSE_CHUNK_MAX_LENGTH=4000 This way, we can avoid errors of exceeding chunk size when using gpt-3.5
Significant-Gravitas · Apr 18, 2023 · e34ede7 · e34ede7
1 parent 67846ba
commit e34ede7
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/autogpt/processing/text.py b/autogpt/processing/text.py
@@ -62,7 +62,7 @@ def summarize_text(
     print(f"Text length: {text_length} characters")
 
     summaries = []
-    chunks = list(split_text(text))
+    chunks = list(split_text(text, CFG.browse_chunk_max_length))
     scroll_ratio = 1 / len(chunks)
 
     for i, chunk in enumerate(chunks):