ScrapeWebsiteTool cannot handle cookies #672
Description
Hi,
I (almost) finished yesterday your presentation course on Deeplearning.ai and I was impressed ;) My first try did not succeed, although.
I am just trying to get an agent to analyze a job posting and give a structured output of the requirements, like in L7_job_application_crew.ipynb from the presentation. I just copied the agent and task:
from crewai_tools import ScrapeWebsiteTool
scrape_tool = ScrapeWebsiteTool()
researcher = Agent(
role="Tech Job Researcher",
goal="Make sure to do amazing analysis on "
"job posting to help job applicants",
tools = [scrape_tool],
verbose=True,
backstory=(
"As a Job Researcher, your prowess in "
"navigating and extracting critical "
"information from job postings is unmatched."
"Your skills help pinpoint the necessary "
"qualifications and skills sought "
"by employers, forming the foundation for "
"effective application tailoring."
)
)
analyze_task = Task(
description=(
"Analyze the job posting URL provided ({job_posting_url}) "
"to extract key skills, experiences, and qualifications "
"required. Use the tools to gather content and identify "
"and categorize the requirements."
),
expected_output=(
"A structured list of job requirements, including necessary "
"skills, qualifications, and experiences."
),
agent=researcher,
# async_execution=True
)
req_crew = Crew(
agents = [researcher],
tasks = [analyze_task],
verbose = True,
full_output = True
)
inputs = {
'job_posting_url': 'https://hu.indeed.com/viewjob?jk=44678430abbc6f69&tk=1hufoopq6ojdt85p&from=serp&vjs=3',
}But when running the crew, the output is:
> Entering new CrewAgentExecutor chain...
I should start by extracting the content of the job posting from the provided URL to analyze the key skills, experiences, and qualifications required.
Action: Read website content
Action Input: {"website_url": "https://hu.indeed.com/viewjob?jk=44678430abbc6f69&tk=1hufoopq6ojdt85p&from=serp&vjs=3"}
Just a moment...Enable JavaScript and cookies to continue
Final Answer: Just a moment...Enable JavaScript and cookies to continue
> Finished chain.
Did it stuck when asked to enable Javascript and cookies?