Integrate old changes

README.md

@@ -9,13 +9,6 @@ To install the package, open your terminal:
 
 ```python
 pip install brightdata-sdk
-```
-> If using macOS, first open a virtual environment for your project
-
-## Quick Start
-
-Create a [Bright Data](https://brightdata.com/cp/setting/) account and copy your API key
-
 ### Initialize the Client
 
 ```python
@@ -25,7 +18,11 @@ client = bdclient(api_token="your_api_token_here") # can also be defined as BRIG
 ```
 
 ### Launch first request
+<<<<<<< HEAD
 Add to your code a serp function
+=======
+Add to your code a serp function.
+>>>>>>> old/main
 ```python
 results = client.search("best selling shoes")
 
@@ -38,10 +35,17 @@ print(client.parse_content(results))
 
 | Feature                        | Functions                   | Description
 |--------------------------|-----------------------------|-------------------------------------
+<<<<<<< HEAD
 | **Scrape every website** | `scrape`                    | Scrape every website using Bright's scraping and unti bot-detection capabilities
 | **Web search**           | `search`                    | Search google and other search engines by query (supports batch searches)
 | **Web crawling**         | `crawl`                     | Discover and scrape multiple pages from websites with advanced filtering and depth control
 | **AI-powered extraction** | `extract`                  | Extract specific information from websites using natural language queries and OpenAI
+=======
+| **Scrape any website** | `scrape`                    | Scrape every website using Bright's scraping and unti bot-detection capabilities
+| **Web search(SERP)**           | `search`                    | Search google and other search engines by query (supports batch searches)
+| **Web crawling**         | `crawl`                     | Discover and scrape multiple pages from websites with advanced filtering and depth control
+| **AI extraction** | `extract`                  | Extract specific information from websites using natural language queries and OpenAI
+>>>>>>> old/main
 | **Content parsing**      | `parse_content`             | Extract text, links, images and structured data from API responses (JSON or HTML)
 | **Browser automation**   | `connect_browser`           | Get WebSocket endpoint for Playwright/Selenium integration with Bright Data's scraping browser
 | **Search chatGPT**       | `search_chatGPT`            | Prompt chatGPT and scrape its answers, support multiple inputs and follow-up prompts
@@ -51,11 +55,19 @@ print(client.parse_content(results))
 | **Client class**         | `bdclient`         | Handles authentication, automatic zone creation and managment, and options for robust error handling
 | **Parallel processing**  | **all functions**  | All functions use Concurrent processing for multiple URLs or queries, and support multiple Output Formats
 
+<<<<<<< HEAD
 ### Try usig one of the functions
 
 #### `Search()`
 ```python
 # Simple single query search
+=======
+### Try using one of the functions
+
+#### `Search()`
+```python
+# Single query search
+>>>>>>> old/main
 result = client.search("pizza restaurants")
 
 # Try using multiple queries (parallel processing), with custom configuration
@@ -69,7 +81,11 @@ results = client.search(
 ```
 #### `scrape()`
 ```python
+<<<<<<< HEAD
 # Simple single URL scrape
+=======
+# Single URL scrape
+>>>>>>> old/main
 result = client.scrape("https://example.com")
 
 # Multiple URLs (parallel processing) with custom options
@@ -83,13 +99,30 @@ results = client.scrape(
 ```
 #### `search_chatGPT()`
 ```python
+<<<<<<< HEAD
 result = client.search_chatGPT(
     prompt="what day is it today?"
     # prompt=["What are the top 3 programming languages in 2024?", "Best hotels in New York", "Explain quantum computing"],
     # additional_prompt=["Can you explain why?", "Are you sure?", ""]  
 )
 
 client.download_content(result) # In case of timeout error, your snapshot_id is presented and you will downloaded it using download_snapshot()
+=======
+# Sync mode (immediate result)
+result = client.search_gpt(
+    prompt="Top startups in Tel Aviv",
+    country="IL",
+    web_search=True
+)
+print(result)
+
+# Async mode (retrieve snapshot later)
+result = client.search_gpt(
+    prompt="Top startups in 2024",
+    sync=False
+)
+print(result["snapshot_id"])
+>>>>>>> old/main
 ```
 
 #### `search_linkedin.`
@@ -125,7 +158,11 @@ print(results) # will print the snapshot_id, which can be downloaded using the d
 result = client.crawl(
     url="https://example.com/",
     depth=2,
+<<<<<<< HEAD
     filter="/product/",           # Only crawl URLs containing "/product/"
+=======
+    include_filter="/product/",           # Only crawl URLs containing "/product/"
+>>>>>>> old/main
     exclude_filter="/ads/",       # Exclude URLs containing "/ads/"
     custom_output_fields=["markdown", "url", "page_title"]
 )
@@ -202,12 +239,23 @@ client.download_content(data)
 ```
 **`download_snapshot`** (for async requests)
 ```python
+<<<<<<< HEAD
 # Save this function to seperate file
 client.download_snapshot("") # Insert your snapshot_id
 ```
 
 > [!TIP]
 > Hover over the "search" or each function in the package, to see all its available parameters.
+=======
+# Save this function to a seperate file
+# Download snapshot
+snapshot_id = "your_snapshot_id_here"  # <-- Replace with your actual snapshot ID
+client.download_snapshot(snapshot_id)  # Insert your snapshot_id
+```
+
+> [!TIP]
+> Hover over the "search" or each function in the package to see all its available parameters.
+>>>>>>> old/main
 
 ![Hover-Over1](https://github.com/user-attachments/assets/51324485-5769-48d5-8f13-0b534385142e)
 
@@ -251,7 +299,11 @@ Discover and scrape multiple pages from websites with advanced filtering.
 - `url`: Single URL string or list of URLs to crawl (required)
 - `ignore_sitemap`: Ignore sitemap when crawling (optional)
 - `depth`: Maximum crawl depth relative to entered URL (optional)
+<<<<<<< HEAD
 - `filter`: Regex to include only certain URLs (e.g. "/product/")
+=======
+- `include_filter`: Regex to include only certain URLs (e.g. "/product/")
+>>>>>>> old/main
 - `exclude_filter`: Regex to exclude certain URLs (e.g. "/ads/")
 - `custom_output_fields`: List of output fields to include (optional)
 - `include_errors`: Include errors in response (default: True)

setup.py

@@ -15,11 +15,14 @@ def read_readme():
 
 # Read version from __init__.py
 def read_version():
-    with open(os.path.join("brightdata", "__init__.py"), "r", encoding="utf-8") as fh:
-        for line in fh:
-            if line.startswith("__version__"):
-                return line.split('"')[1]
-    return "1.0.0"
+    try:
+        with open(os.path.join("src", "brightdata", "__init__.py"), "r", encoding="utf-8") as fh:
+            for line in fh:
+                if line.startswith("__version__"):
+                    return line.split('"')[1]
+    except FileNotFoundError:
+        return "0.1.0"
+    return "0.1.0"
 
 setup(
     name="brightdata-sdk",
@@ -30,27 +33,15 @@ def read_version():
     long_description=read_readme(),
     long_description_content_type="text/markdown",
     url="https://github.com/brightdata/brightdata-sdk-python",
-    packages=find_packages(),
-    classifiers=[
-        "Development Status :: 4 - Beta",
-        "Intended Audience :: Developers",
-        "License :: OSI Approved :: MIT License",
-        "Operating System :: OS Independent",
-        "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8", 
-        "Programming Language :: Python :: 3.9",
-        "Programming Language :: Python :: 3.10",
-        "Programming Language :: Python :: 3.11",
-        "Programming Language :: Python :: 3.12",
-        "Topic :: Internet :: WWW/HTTP",
-        "Topic :: Software Development :: Libraries :: Python Modules",
-        "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
-    ],
+    package_dir={"": "src"},
+    packages=find_packages(where="src"),
     python_requires=">=3.7",
     install_requires=[
         "requests>=2.25.0",
         "python-dotenv>=0.19.0",
+        "aiohttp>=3.8.0",
+        "beautifulsoup4>=4.9.0",
+        "openai>=1.0.0",
     ],
     extras_require={
         "dev": [
@@ -61,10 +52,10 @@ def read_version():
             "flake8>=3.8.0",
         ],
     },
-    keywords="brightdata, web scraping, proxy, serp, api, data extraction",
+    keywords="brightdata, web scraping, proxy, serp, api, data extraction, aiohttp, bs4, openai",
     project_urls={
         "Bug Reports": "https://github.com/brightdata/brightdata-sdk-python/issues",
         "Documentation": "https://github.com/brightdata/brightdata-sdk-python#readme",
         "Source": "https://github.com/brightdata/brightdata-sdk-python",
     },
-)
+)

src/__init__.py

@@ -0,0 +1,82 @@
+"""
+## Bright Data SDK for Python
+
+A comprehensive SDK for Bright Data's Web Scraping and SERP APIs, providing
+easy-to-use methods for web scraping, search engine result parsing, and data management.
+## Functions:
+First import the package and create a client:
+```python
+from brightdata import bdclient
+client = bdclient(your-apy-key)
+```
+Then use the client to call the desired functions:  
+#### scrape()
+- Scrapes a website using Bright Data Web Unblocker API with proxy support (or multiple websites sequentially)
+- syntax: `results = client.scrape(url, country, max_workers, ...)`
+#### .scrape_linkedin. class
+- Scrapes LinkedIn data including posts, jobs, companies, and profiles, recieve structured data as a result
+- syntax: `results = client.scrape_linkedin.posts()/jobs()/companies()/profiles() # insert parameters per function`
+#### search()
+- Performs web searches using Bright Data SERP API with customizable search engines (or multiple search queries sequentially)
+- syntax: `results = client.search(query, search_engine, country, ...)`
+#### .search_linkedin. class
+- Search LinkedIn data including for specific posts, jobs, profiles. recieve the relevent data as a result
+- syntax: `results = client.search_linkedin.posts()/jobs()/profiles() # insert parameters per function`
+#### search_chatGPT()
+- Interact with ChatGPT using Bright Data's ChatGPT API, sending prompts and receiving responses
+- syntax: `results = client.search_chatGPT(prompt, additional_prompt, max_workers, ...)`
+#### download_content() / download_snapshot()
+- Saves the scraped content to local files in various formats (JSON, CSV, etc.)
+- syntax: `client.download_content(results)`
+- syntax: `client.download_snapshot(results)`
+#### connect_browser()
+- Get WebSocket endpoint for connecting to Bright Data's scraping browser with Playwright/Selenium
+- syntax: `endpoint_url = client.connect_browser()` then use with browser automation tools
+#### crawl()
+- Crawl websites to discover and scrape multiple pages using Bright Data's Web Crawl API
+- syntax: `result = client.crawl(url, filter, exclude_filter, depth, ...)`
+#### parse_content()
+- Parse and extract useful information from API responses (JSON or HTML)
+- syntax: `parsed = client.parse_content(data, extract_text=True, extract_links=True)`
+
+### Features:
+- Web Scraping: Scrape websites using Bright Data Web Unlocker API with proxy support
+- Search Engine Results: Perform web searches using Bright Data SERP API  
+- Web Crawling: Discover and scrape multiple pages from websites with advanced filtering
+- Content Parsing: Extract text, links, images, and structured data from API responses
+- Browser Automation: Simple authentication for Bright Data's scraping browser with Playwright/Selenium
+- Multiple Search Engines: Support for Google, Bing, and Yandex
+- Parallel Processing: Concurrent processing for multiple URLs or queries
+- Robust Error Handling: Comprehensive error handling with retry logic
+- Input Validation: Automatic validation of URLs, zone names, and parameters
+- Zone Management: Automatic zone creation and management
+- Multiple Output Formats: JSON, raw HTML, markdown, and more
+"""
+
+from .client import bdclient
+from .exceptions import (
+    BrightDataError,
+    ValidationError,
+    AuthenticationError,
+    ZoneError,
+    NetworkError,
+    APIError
+)
+from .utils import parse_content, parse_multiple, extract_structured_data
+
+__version__ = "1.1.3"
+__author__ = "Bright Data"
+__email__ = "support@brightdata.com"
+
+__all__ = [
+    'bdclient',
+    'BrightDataError',
+    'ValidationError', 
+    'AuthenticationError',
+    'ZoneError',
+    'NetworkError',
+    'APIError',
+    'parse_content',
+    'parse_multiple',
+    'extract_structured_data'
+]

src/api/__init__.py

@@ -0,0 +1,13 @@
+from .scraper import WebScraper
+from .search import SearchAPI
+from .chatgpt import ChatGPTAPI
+from .linkedin import LinkedInAPI
+from .crawl import CrawlAPI
+
+__all__ = [
+    'WebScraper',
+    'SearchAPI',
+    'ChatGPTAPI',
+    'LinkedInAPI',
+    'CrawlAPI'
+]