support exclusion of directories from scanning for pipeline specs

[OriHoch]

All sub-directories are scanned for pipeline-specs, this is not optimal, especially with data directories contained under the project root directory.

Added support for exclusion of directory patterns (glob), by default, excluding .* directories and allowing to exclude additional patterns.

Uses os.walk support for modifying the directories that will be scanned by modifying the dirnames list in-place.

See updated README for usage details

[coveralls]

Pull Request Test Coverage Report for Build 1002

• 70 of 188 (37.23%) changed or added relevant lines in 2 files are covered.
• 45 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.3%) to 17.568%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
datapackage_pipelines/utilities/dirtools.py	66	184	35.87%

Files with Coverage Reduction	New Missed Lines	%
datapackage_pipelines/specs/specs.py	1	79.49%
.tox/py36-plyvel/lib/python3.6/site-packages/datapackage_pipelines/specs/specs.py	22	0.0%
.tox/py36-sqlite/lib/python3.6/site-packages/datapackage_pipelines/specs/specs.py	22	0.0%

Totals
Change from base Build 1000:	-0.3%
Covered Lines:	1731
Relevant Lines:	9853

---

💛 - Coveralls

[brew]

Could this support an ignore file (like .dockerignore, .gitignore etc)? Perhaps .dpp_spec_ignore? Files to ignore seems like something you'd always want to do for all installations of a project, rather than something to configure for a specific installation in an env var.

[OriHoch]

thanks @brew yes, makes sense, I'll modify it

[OriHoch]

updated to use .dpp_spec_ignore file

[akariv]

@OriHoch can you please check why ci is failing?

[OriHoch]

@akariv fixed