Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closes #30: Crawler unit tests #36

Merged
merged 5 commits into from
Oct 14, 2017
Merged

Closes #30: Crawler unit tests #36

merged 5 commits into from
Oct 14, 2017

Conversation

bobheadxi
Copy link
Member

@bobheadxi bobheadxi commented Oct 14, 2017 •
edited

Related Issue

#30

Description

  • Unit tests and test data for UbcBroadCrawler
  • Unit tests for SolrPipeline
  • Updates to import paths and requirements.txt get crawler working with new directory structure from Solr queries and better Docker+Travis setup #33
  • Shinier ZenHub badge on README

WIKI Updates

  • Slight update to Schemas (specifying DT format)

Todos

  • Prevent unintended Solr code coverage
  • More in-depth tests of UbcBroadCrawler ie url extractions

General:

  • Tests
  • Wiki

@coveralls
Copy link

Coverage Status

Coverage increased (+41.5%) to 45.417% when pulling 083da4e on 30-crawler-tests into 2f691a4 on master.

bfbachmann
bfbachmann approved these changes Oct 14, 2017
View reviewed changes
Copy link
Member

@bfbachmann bfbachmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLEAN 🎓

sleuth_crawler/tests/mocks.py
@@ -0,0 +1,23 @@
from scrapy.http import Request, TextResponse

def mock_response(file_name, url=None):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

@bobheadxi bobheadxi merged commit b8f6e00 into master Oct 14, 2017
@bobheadxi bobheadxi deleted the 30-crawler-tests branch October 14, 2017 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bfbachmann bfbachmann bfbachmann approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@bobheadxi @coveralls @bfbachmann