Request guideline

[trung1704ptit]

Hi,
Thanks for your sharing repo. But I cannot run crawl indeed job. I already run by docker. When I see the Flower screen it created a job success but nothing happen. The question is how do I know it works.

[yennanliu]

hi @trung1704ptit
thx for the feedback! You may need to run indeed scrapping manually.
Kindly try the below commands mentioned in readme

https://github.com/yennanliu/web_scraping

# "indeed_scrap_task" task
$ curl -X POST  http://localhost:5555/api/task/async-apply/tasks.indeed_scrap_task

# "indeed_scrap_api_V1" task
$ curl -X POST -d '{"args":["New+York"]}' http://localhost:5555/api/task/async-apply/tasks.indeed_scrap_api_V1

Cheers,

[yennanliu]

you can also refer below stand-alone py script, it should be able to execute indeed scrapping as well
https://github.com/yennanliu/web_scraping/blob/master/cron_indeed_scrapping_test.py

[trung1704ptit]

@yennanliu Thank you, I have already run it. But need to update the code to crawl the job information because indeed updated the format HTML.

[trung1704ptit]

and more question is if I crawl job successfully, how do I see the result on MongoDB. I don't see the way to see the result

[yennanliu]

@yennanliu Thank you, I have already run it. But need to update the code to crawl the job information because indeed updated the format HTML.

yes, you may have to update the code when Indeed updates its HTML.
A PR is welcomed!

[yennanliu]

and more question is if I crawl job successfully, how do I see the result on MongoDB. I don't see the way to see the result

Hi @trung1704ptit
Not sure if you can find such information in Redis instead. (Redis as CELERY_RESULT_BACKEND)
MongoDB is not used in this service actually.
https://github.com/yennanliu/web_scraping/blob/master/docker-compose.yml#L28
https://github.com/yennanliu/web_scraping/blob/master/Dockerfile#L7