-
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sweep: provide me all css selectors for content for selectolax #10
Comments
🚀 Here's the PR! #12See Sweep's progress at the progress dashboard! 💎 Sweep Pro: I'm using GPT-4. You have unlimited GPT-4 tickets. (tracking ID:
4eab7df4bc )Tip I'll email you at hardeep.ex@gmail.com when I complete this pull request! Actions (click)
Sandbox Execution ✓Here are the sandbox execution logs prior to making any changes: Sandbox logs for
|
}, | |
extra_preprocessors=[CSS("div.list_item")], | |
) | |
# Define the SchemaScraper for the single deal pages | |
deal_scraper = SchemaScraper( | |
{ | |
"title": "str", | |
"url": "url", | |
"price": "float", | |
"regular_price": "float", | |
"details": "str", | |
}, |
scrapegost/tests/live/test_live.py
Lines 55 to 59 in f0a2d7b
actor_schema = { | |
"actor": "string", | |
"image": "url", | |
"roles": {"name": "string", "character": "string"}, |
Lines 65 to 68 in f0a2d7b
If you visit the page <https://comedybangbang.fandom.com/wiki/Operation_Golden_Orb> viewing the source will reveal that all of the interesting content is in an element `<div id="content" class="page-content">`. | |
Just as we might if we were writing a real scraper, we'll write a CSS selector to grab this element, `div.page-content` will do. |
I also found the following external resources that might be helpful:
Summaries of links found in the content:
https://w.dam-img.rfdcontent.com/offers/013/736/864/200x200_pad.jpg:
The page contains a list of deals from different merchants. The first deal is a sponsored offer for a Samsung 77" OLED 4K Smart TV, which is $1000 off. The offer includes a description of the TV's features such as deep blacks, clean whites, and full shades of lively colors. The second deal is from Kitchen Stuff Plus, offering 50% off the Ballarini Bologna Non-Stick Wok & Frypan Set and more. The third deal is from the PlayStation Store, offering free monthly games for PlayStation Plus members. The page also includes pagination for navigating through the deals.
https://q.dam-img.rfdcontent.com/offers/013/736/863/100x100_pad.jpg:
The page contains a list of deals from different merchants. The first deal is from Kitchen Stuff Plus, offering 50% off on the Ballarini Bologna Non-Stick Wok & Frypan Set. The second deal is from PlayStation Store, offering free monthly games for PlayStation Plus members. The page also includes pagination for navigating through the deals.
https://p.dam-img.rfdcontent.com/offers/013/736/864/100x100_pad.jpg:
The page contains information about two deals: a $1000 off 77" OLED 4K Smart TV from Samsung and 50% off Ballarini Bologna Non-Stick Wok & Frypan Set from Kitchen Stuff Plus. The Samsung TV features OLED technology with deep blacks, clean whites, and full shades of lively colors. It also has 4K AI upscaling and a Dolby Atmos experience. The Kitchen Stuff Plus deal includes various red hot deals. The page also includes pagination for more deals and a footer with additional information.
https://t.dam-img.rfdcontent.com/offers/013/736/863/100x100_pad.jpg:
The page contains a list of deals from different merchants. The first deal is from Kitchen Stuff Plus, offering 50% off on the Ballarini Bologna Non-Stick Wok & Frypan Set. The second deal is from PlayStation Store, offering free monthly games for PlayStation Plus members. The page also includes pagination for navigating through the deals.
https://tpc.googlesyndication.com/simgad/461817906561256968:
The page contains a list of deals from different merchants. The first deal is a $1000 off 77" OLED 4K Smart TV from Samsung. The deal includes a description of the TV's features such as deep blacks, clean whites, and full shades of lively colors. The second deal is about free monthly games for PlayStation Plus subscribers. The games mentioned are A Plague Tale Requiem and Evil West. The page also includes pagination for navigating through the deals.
https://h.dam-img.rfdcontent.com/offers/013/736/863/200x200_pad.jpg:
The page contains a list of deals from different merchants. The first deal is a sponsored offer for a $1000 discount on a 77" OLED 4K Smart TV from Samsung. The second deal is from Kitchen Stuff Plus, offering 50% off on a Ballarini Bologna Non-Stick Wok & Frypan Set. The third deal is from PlayStation Store, offering free monthly games for PlayStation Plus members. The page also includes pagination for navigating through the deals.
https://o.dam-img.rfdcontent.com/offers/013/736/864/100x100_pad.jpg:
The page contains information about two deals: a $1000 off 77" OLED 4K Smart TV from Samsung and 50% off Ballarini Bologna Non-Stick Wok & Frypan Set from Kitchen Stuff Plus. The Samsung TV features OLED technology with deep blacks, clean whites, and full shades of lively colors. It also has 4K AI upscaling and a Dolby Atmos experience. The Kitchen Stuff Plus deal includes various red hot deals. The page also includes pagination for more deals and a footer with additional information.
Step 2: ⌨️ Coding
Modify docs/examples/tutorial/redflagdeals_scraper.py with contents:
• Modify the SchemaScraper definition to include the additional fields the user wants to scrape. The modified schema should look like this: { "title": "str", "url": "url", "image": "url", "description": "str", "price": "float", "regular_price": "float", "details": "str", }
• Add the appropriate CSS selectors to the extra_preprocessors list to target the correct elements on the webpage. The selectors should target the elements containing the title, URL, image, and description for each deal. The modified extra_preprocessors list should look like this: extra_preprocessors=[CSS("div.list_item"), CSS("a.offer_image"), CSS("h2.offer_title"), CSS("p.offer_description")]--- +++ @@ -11,7 +11,7 @@ "dealer": "str", "comments_count": "int", }, - extra_preprocessors=[CSS("div.list_item")], + extra_preprocessors=[CSS("div.list_item"), CSS("a.offer_image"), CSS("h2.offer_title"), CSS("p.offer_description")], ) # Define the SchemaScraper for the single deal pages @@ -19,6 +19,8 @@ { "title": "str", "url": "url", + "image": "url", + "description": "str", "price": "float", "regular_price": "float", "details": "str",
- Running GitHub Actions for
docs/examples/tutorial/redflagdeals_scraper.py
✓ Edit
Check docs/examples/tutorial/redflagdeals_scraper.py with contents:Ran GitHub Actions for a53e451d9659b74837cc2291a418d7802299642b:
Create tests/live/test_new_scraper.py with contents:
• Create a new test case to ensure that the modified scraper correctly extracts the desired data from the webpage.
• Import the necessary libraries and modules at the beginning of the file. This should include unittest and the modified scraper from the redflagdeals_scraper.py file.
• Define a new class for the test case that inherits from unittest.TestCase.
• Within this class, define a new method for the test case. This method should use the modified scraper to scrape data from a test webpage that contains the same structure as the actual webpage the user wants to scrape.
• The test case should assert that the scraped data matches the expected data for each field in the schema. The expected data should be manually defined within the test case based on the content of the test webpage.
- Running GitHub Actions for
tests/live/test_new_scraper.py
✓ Edit
Check tests/live/test_new_scraper.py with contents:Ran GitHub Actions for de65ed7faa208042b58c260aa323d2e2a4f36141:
Step 3: 🔁 Code Review
I have finished reviewing the code for completeness. I did not find errors for sweep/provide_me_all_css_selectors_for_content_1
.
🎉 Latest improvements to Sweep:
- We just released a dashboard to track Sweep's progress on your issue in real-time, showing every stage of the process – from search to planning and coding.
- Sweep uses OpenAI's latest Assistant API to plan code changes and modify code! This is 3x faster and significantly more reliable as it allows Sweep to edit code and validate the changes in tight iterations, the same way as a human would.
- Try using the GitHub issues extension to create Sweep issues directly from your editor! GitHub Issues and Pull Requests.
💡 To recreate the pull request edit the issue title or description. To tweak the pull request, leave a comment on the pull request.
Join Our Discord
sweep: how it will work like for example i need for this url https://www.redflagdeals.com/deals/ |
$1000 off 77” OLED 4K Smart TV S95C
Samsung’s unique OLED provides deep blacks, clean whites and full shades of lively colours. With cutting-edge, self-illuminating pixels driven by Quantum Dot, you can enjoy an accurate, pure and bright picture.
Kitchen Stuff Plus
See the Best Red Hot Deals from Kitchen Stuff Plus
PlayStation Store
PlayStation's Free Monthly Games for January!
Displaying 1–20 of 260 Deals
Checklist
docs/examples/tutorial/redflagdeals_scraper.py
✓ a53e451 Editdocs/examples/tutorial/redflagdeals_scraper.py
✓ Edittests/live/test_new_scraper.py
✓ de65ed7 Edittests/live/test_new_scraper.py
✓ EditThe text was updated successfully, but these errors were encountered: