-
Have python and pip working
-
Install virtualenv ( https://virtualenv.pypa.io/en/stable/installation/ )
-
make a directory where your project will run
mkdir p3-gpi-export
-
Tell it to run in a vritualenv
virtualenv p3-gpi-export
-
Go in the directory
cd p3-gpi-export
-
Start the virtualenv
source bin/activate
-
Install scrapy
pip install Scrapy
-
Install dateparser
pip install dateparser
-
Initiate a scrapy project
scrapy startproject p3_gpi_export
-
Download the crawler
git clone https://github.com/greenpeace/planet4-gpi-export
-
Copy the crawler file in the scrapy project crawlers
cp planet4-gpi-export/all_spider.py p3_gpi_export/p3_gpi_export/spiders/all_spider.py
-
Go in the directory of the scrapy project
cd p3_gpi_export/
-
Run the crawler
scrapy crawl all
-
Notifications
You must be signed in to change notification settings - Fork 2
License
greenpeace/planet4-gpi-export
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published