-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Breaking down large files into smaller chunks based on context window size #4
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No changes required. The modifications made in the files gpt_migrate/steps/debug.py
, gpt_migrate/steps/migrate.py
, gpt_migrate/steps/test.py
, and gpt_migrate/utils.py
are well implemented. The addition of chunk processing logic to handle large files is a good approach. Keep up the good work!
Sweep: Revert the accidental deletions in the file |
gpt_migrate/steps/debug.py
Outdated
waiting_message=f"Planning actions for debugging...", | ||
success_message="", | ||
globals=globals) | ||
# Split the error message into chunks based on the context window size for more efficient processing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sweep: Don't run chunking in this one at all
success_message=f"Created {testfile}.tests.py file in directory gpt_migrate.", | ||
globals=globals) | ||
return f"{testfile}.tests.py" | ||
def create_tests(testfile,globals): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sweep: Don't run chunking in this one at all
Description
This PR addresses the issue #1 by introducing a new function
split_file_into_chunks
ingpt_migrate/utils.py
that splits large files into smaller chunks based on a specified context window size.Changes Made
split_file_into_chunks(filepath, chunk_size)
ingpt_migrate/utils.py
that reads a file, splits it into chunks of the specified size, and returns a list of these chunks.split_file_into_chunks
:llm_write_file
andllm_write_files
ingpt_migrate/utils.py
write_migration
ingpt_migrate/steps/migrate.py
debug_error
anddebug_testfile
ingpt_migrate/steps/debug.py
create_tests
,validate_tests
, andrun_test
ingpt_migrate/steps/test.py
Testing
Checklist
Fixes #1.
To checkout this PR branch, run the following command in your terminal: