daf-ckan-crawler

Pseudo distributed ckan crawler

For mac users

export LC_ALL=en_US.UTF-8

export LANG=en_US.UTF-8

install pipenv

pipenv install pipenv shell

python local_crawler.py

Name		Name	Last commit message	Last commit date
Latest commit History 13 Commits
.idea		.idea
dataschema		dataschema
old_script		old_script
.DS_Store		.DS_Store
.gitignore		.gitignore
LICENSE		LICENSE
Pipfile		Pipfile
Pipfile.lock		Pipfile.lock
README.md		README.md
hdfs_master_crawler.py		hdfs_master_crawler.py
hdfs_worker_crawler.py		hdfs_worker_crawler.py
json_template.json		json_template.json
local_crawler.py		local_crawler.py
test_dati_gov_ckan.json		test_dati_gov_ckan.json

Provide feedback