Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pre/beta #371

Merged
merged 153 commits into from
Jun 17, 2024
Merged

Pre/beta #371

merged 153 commits into from
Jun 17, 2024

Conversation

PeriniM
Copy link
Collaborator

@PeriniM PeriniM commented Jun 12, 2024 •
edited
Loading

🚀 New Features and Enhancements

Major Changes:

  1. 📦 Pydantic Schema Support:

    • Integrated Pydantic schema support in both graphs and nodes to enhance data validation and management.

  2. 📄 Multi Scrapers:

    • Added multi scrapers for various file formats including PDF, CSV, JSON, and XML, enabling more versatile data extraction capabilities.

  3. 📝 ScriptCreatorGraph Enhancements:

    • Updated ScriptCreatorGraph to generate Python scripts that adhere to the Pydantic schema for better structured and validated outputs.

  4. 🔗 ScriptCreatorMultiGraph:

    • Introduced ScriptCreatorMultiGraph, which creates a single Python script capable of scraping multiple URLs, streamlining the multi-source data collection process.

  5. ⚡ Semantic Chunking:

    • Implemented semchunk for faster and more efficient semantic chunking, improving the performance and accuracy of text segmentation.

  6. 💾 Caching Embeddings:

    • Added a cache parameter to store embeddings for a specific source, allowing for reuse and reducing the need for repeated computations.

  7. 📊 Telemetry Support:

    • Added telemetry support to help us keep improving the features the community likes the most.

  8. 🌐 International Language Support:

    • Expanded README to include support for Chinese, Japanese, Korean, and Russian, making the documentation more accessible to a global audience.

Minor Changes:

  • General code cleanup and optimization for better performance and maintainability.

I thank all the contributors for the release listed as "Participants" in this PR!

🕷️🕷️🕷️🕷️🕷️🕷️🕷️🕷️🕷️🕷️🕷️

VinciGit00 and others added 30 commits May 29, 2024 12:14
@VinciGit00
Merge pull request #314 from VinciGit00/main
b553602 
reallignment
@VinciGit00
fix: oneapi model
4fcb990
@semantic-release-bot
ci(release): 1.5.3-beta.1 [skip ci]
6ea1d2c 
## [1.5.3-beta.1](v1.5.2...v1.5.3-beta.1) (2024-05-29)

### Bug Fixes

* oneapi model ([4fcb990](4fcb990))
@VinciGit00
removed unused file
1aa8c86
fix: typo in prompt
4639f0c
@PeriniM
Merge pull request #319 from jmfk/pre/beta
e734830 
fix: typo in prompt
@semantic-release-bot
ci(release): 1.5.3-beta.2 [skip ci]
b57bcef 
## [1.5.3-beta.2](v1.5.3-beta.1...v1.5.3-beta.2) (2024-05-30)

### Bug Fixes

* typo in prompt ([4639f0c](4639f0c))
@VinciGit00
Create chinese.md
cdba5ef
@VinciGit00
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai…
6d1d91a 
… into pre/beta
@VinciGit00
feat: removed rag node
930f673
@VinciGit00
Merge branch 'pre/beta' into temp
25352a5
@VinciGit00
Merge pull request #320 from VinciGit00/temp
25de33e 
Alignment
@semantic-release-bot
ci(release): 1.5.5-beta.1 [skip ci]
38d138e 
## [1.5.5-beta.1](v1.5.4...v1.5.5-beta.1) (2024-05-31)

### Bug Fixes

* oneapi model ([4fcb990](4fcb990))
* typo in prompt ([4639f0c](4639f0c))

### CI

* **release:** 1.5.3-beta.1 [skip ci] ([6ea1d2c](6ea1d2c))
* **release:** 1.5.3-beta.2 [skip ci] ([b57bcef](b57bcef))
@VinciGit00
feat: add pdf scraper multi graph
f5cbd80
@VinciGit00
add example
4d42d7b
@VinciGit00
feat: add json multiscraper
5bda918
@VinciGit00
add rag node
fff1232
@VinciGit00
add openai and oneapi examples
1fe4975
@VinciGit00
feat: add forcing format as json
5cfc101
@semantic-release-bot
ci(release): 1.6.0-beta.1 [skip ci]
1d217e4 
## [1.6.0-beta.1](v1.5.5-beta.1...v1.6.0-beta.1) (2024-06-02)

### Features

* add forcing format as json ([5cfc101](5cfc101))
@VinciGit00
add examples
fa9722d
@VinciGit00
feat: add csv scraper and xml scraper multi
b408655
@VinciGit00
add all possible examples
743dfe1
@PeriniM
Merge pull request #323 from VinciGit00/refactoring-pdf_scraper
79ace11 
Refactoring pdf scraper and json scrape
@semantic-release-bot
ci(release): 1.6.0-beta.2 [skip ci]
ed1dc0b 
## [1.6.0-beta.2](v1.6.0-beta.1...v1.6.0-beta.2) (2024-06-03)

### Features

* add csv scraper and xml scraper multi ([b408655](b408655))
* add json multiscraper ([5bda918](5bda918))
* add pdf scraper multi graph ([f5cbd80](f5cbd80))
* removed rag node ([930f673](930f673))
@VinciGit00
add new examples
1dde43c
@VinciGit00
feat: removed a bug
8de720d
@semantic-release-bot
ci(release): 1.6.0-beta.3 [skip ci]
b70cb37 
## [1.6.0-beta.3](v1.6.0-beta.2...v1.6.0-beta.3) (2024-06-03)

### Features

* removed a bug ([8de720d](8de720d))
@VinciGit00
feat: fix an if
c8d556d
@VinciGit00
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai…
f36dd8b 
… into pre/beta
semantic-release-bot and others added 2 commits June 16, 2024 11:35
@semantic-release-bot
ci(release): 1.7.0-beta.8 [skip ci]
a87702f 
## [1.7.0-beta.8](v1.7.0-beta.7...v1.7.0-beta.8) (2024-06-16)

### Bug Fixes

* shallow copy config of create_embedder ([62b372b](62b372b))

### Refactor

* add missing schemas and renamed files ([09cb6e9](09cb6e9))
@VinciGit00
fix: fix robot node
2419003
shubihu and others added 3 commits June 17, 2024 16:26
@shubihu
Update abstract_graph.py
d8d5cd2 
fix: Incorrect API Key Error with OpenAI Proxy
@VinciGit00
Merge branch 'pre/beta' into main
da93162
@VinciGit00
Merge pull request #387 from shubihu/main
d7cd1df 
fix: Incorrect API Key Error with OpenAI Proxy
semantic-release-bot and others added 3 commits June 17, 2024 09:04
@semantic-release-bot
ci(release): 1.7.0-beta.10 [skip ci]
7f3b907 
## [1.7.0-beta.10](v1.7.0-beta.9...v1.7.0-beta.10) (2024-06-17)

### Bug Fixes

* removed duplicate from ollama dictionary ([dcd216e](dcd216e))

### CI

* **release:** 1.6.1 [skip ci] ([44fbd71](44fbd71))
@VinciGit00
add smart_scraper_openai_test
6a753f2
@VinciGit00
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai…
93342b4 
… into pre/beta
semantic-release-bot and others added 3 commits June 17, 2024 12:08
@semantic-release-bot
ci(release): 1.7.0-beta.11 [skip ci]
c016efd 
## [1.7.0-beta.11](v1.7.0-beta.10...v1.7.0-beta.11) (2024-06-17)

### Features

* **telemetry:** add telemetry module ([080a318](080a318))

### Docs

* refactor graph section and added telemetry ([39bf4c9](39bf4c9))
@VinciGit00
fix: add chinese embedding model
03ffebc
@VinciGit00
Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai…
48ff87f 
… into pre/beta
@semantic-release-bot
ci(release): 1.7.0-beta.12 [skip ci]
a794405 
## [1.7.0-beta.12](v1.7.0-beta.11...v1.7.0-beta.12) (2024-06-17)

### Bug Fixes

* add chinese embedding model ([03ffebc](03ffebc))
@PeriniM PeriniM requested a review from VinciGit00 June 17, 2024 12:50
@VinciGit00 VinciGit00 merged commit 4238210 into main Jun 17, 2024
6 checks passed
@github-actions GitHub Actions
Copy link

🎉 This PR is included in version 1.7.0-beta.13 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@VinciGit00 VinciGit00 Awaiting requested review from VinciGit00

Assignees
No one assigned
Labels
released on @dev
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@PeriniM @VinciGit00 @semantic-release-bot @seyf97 @elijahbenizzy @iamgodot @tejhande @supercoder-dev @Kshitij-Jande @shubihu