Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Publish #15

Merged
merged 6 commits into from
Jan 21, 2024
Merged

Publish #15

merged 6 commits into from
Jan 21, 2024

Conversation

Daethyra
Copy link
Owner

@Daethyra Daethyra commented Jan 21, 2024 •
edited
Loading

  • refactor context_converter
  • changed pypi publish branch
  • updated license
  • updated readme
  • added speed test results
  • changed some parameter values in converter.py and main.py

@Daethyra
refactor(context_converter): adjust batch sizes for better performance
ce26483 
Changed batch sizes in the HTMLToMarkdownConverter class to improve processing efficiency and reduce memory consumption. Reduced batch_size from 32 to 16 in _process_embeddings method and from 512 to 256 in main function for improved performance.
@Daethyra
Changed pypi-publish branch to 'publish' from 'main' to avoid spamming
6ba80c8
@Daethyra
chore(license): update license to MIT and year
c6c09aa 
Updated the LICENSE file to reflect a change in licensing to MIT and updated copyright year to 2024. This change was made to align with the project's licensing requirements and to ensure accurate copyright information.
@Daethyra
feat(runtime-speed-test): add speed test results and update version t…
e268acf 
…o 1.1.0

Added runtime speed test results for different configurations of chunk size and batch processing, along with an update to the project version in pyproject.toml. This change provides insights into performance optimization and allows users to understand the impact of configuration changes on processing time.
@Daethyra
refactor(converter): improve batch processing and redundancy threshold
44ad798 
Refactored the _process_embeddings method in HTMLToMarkdownConverter to use batch processing for improved efficiency. Also adjusted the redundancy threshold from 1.0 to 0.86899 for better filtering of similar lines.
@Daethyra Daethyra added documentation Improvements or additions to documentation enhancement New feature or request labels Jan 21, 2024
@Daethyra Daethyra self-assigned this Jan 21, 2024
@Daethyra Daethyra merged commit 0150ca8 into main Jan 21, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@Daethyra Daethyra

Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
No open projects
Empower Others to Build RAG-AI Applications
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Daethyra