Permalink
Commits on Nov 7, 2017
  1. Merge pull request #97 from DeanSherwin/get-scrapy-port

    holgerd77 committed Nov 7, 2017
    Get scrapy port
  2. Merge pull request #96 from DeanSherwin/patch-1

    holgerd77 committed Nov 7, 2017
    Note to remind users to add checkers module to settings
Commits on Oct 9, 2017
Commits on Aug 14, 2017
Commits on Jun 29, 2017
  1. Release commit for v.0.13.0

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 29, 2017
Commits on Jun 28, 2017
  1. Fixed bug calling _do_req_info_replacements in scraper with wrong ord…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 28, 2017
    …er of params leading (obviously :-)) to unpredicted behaviour
Commits on Jun 26, 2017
  1. Minor

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 26, 2017
  2. Added num_pages_follow|npf as possible command line parameter for FOL…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 26, 2017
    …LOW pagination
  3. Respect max_items_read parameter for FOLLOW pagination

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 26, 2017
  4. Fixed another FOLLOW pagination bug

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 26, 2017
  5. Fixed pagination bugs

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 26, 2017
  6. Fixed bug wrongly checking for pagination_page_replace on FOLLOW pagi…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 26, 2017
    …nation
Commits on Jun 23, 2017
  1. Added extra XPath for follow pagination to also extract the page numb…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 23, 2017
    …er/name, new {follow_page} placeholder, new migration 0025
Commits on Jun 20, 2017
  1. Replaced JSONDecodeError with ValueError

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 20, 2017
  2. Added item._dds_id_str attribute for easier, more consistent item ref…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 20, 2017
    …erencing, refactoring of existing references
  3. Fixed bugs with non-string scraped value, page number not being incre…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 20, 2017
    …mented in start_requests method
Commits on Jun 16, 2017
  1. Allowing/enabling page placeholders on attributes

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 16, 2017
  2. Allow using main page RPT for followed pages if no follow page RPT is…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 16, 2017
    … defined
  3. Some refactoring on the django_spider class

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 16, 2017
Commits on Jun 14, 2017
  1. First draft of follow page_type implementation

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 14, 2017
  2. Added follow_pages_by_xpath and num_pages_follow attributes to scrape…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 14, 2017
    …r model, new FOLLOW_PAGE type choice for RPTs, new migrations 0023,0024
  3. Dropped support for Scrapy 1.1, 1.2 and 1.3 (Scrapy 1.4 only supporte…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 14, 2017
    …d version)
  4. Using response.follow function from Scrapy 1.4 for following detail p…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 14, 2017
    …age URL links (supports relative URLs)
  5. Allow/enable {page} placeholders for detail page request info fields

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 14, 2017
Commits on Jun 13, 2017
  1. Added new option UNRESOLVED to scraper work status, new migration 0022

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 13, 2017
  2. Changed the event used for loading the custom Django admin JS code to…

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 13, 2017
    … fix resizing of scraper elem form fields not working
  3. Added a general settings tab for the scraper form in the Django admin

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 13, 2017
  4. Make loading of JSON main page content more robust in error case

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 13, 2017
  5. Output DDS configuration dict on DEBUG log level

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 13, 2017
  6. Added short forms for CL options (e.g. 'sp' for 'start_page',...)

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 13, 2017
Commits on Jun 12, 2017
  1. Updated open_news example fixture

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 12, 2017
  2. Release commit for v.0.12.4

    holgerd77 holgerd77
    holgerd77 authored and holgerd77 committed Jun 12, 2017