Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Implement Scrapy source #320

Closed
wants to merge 55 commits into from
Closed

[WIP] Implement Scrapy source #320

wants to merge 55 commits into from

Commits on Jan 18, 2024

  1. Implement Scrapy source

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    22806fa View commit details
    Browse the repository at this point in the history

  2. Cleanup

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    7f9eb88 View commit details
    Browse the repository at this point in the history

  3. Update docstrings

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    c34ed4f View commit details
    Browse the repository at this point in the history

  4. Update docstrings

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    c540d8c View commit details
    Browse the repository at this point in the history

  5. Rename pipeline

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    52b5c68 View commit details
    Browse the repository at this point in the history

  6. Add scrapy dependencies

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    b4100db View commit details
    Browse the repository at this point in the history

  7. Fix linting issues

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    ba7b257 View commit details
    Browse the repository at this point in the history

  8. Fix test issues

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    36362a6 View commit details
    Browse the repository at this point in the history

  9. Re-arrange arguments

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    38f8c8a View commit details
    Browse the repository at this point in the history

  10. Add test directory

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    e149186 View commit details
    Browse the repository at this point in the history

  11. Add basic test

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    68459b9 View commit details
    Browse the repository at this point in the history

  12. Join crawler process

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    2785b14 View commit details
    Browse the repository at this point in the history

  13. Move spider example and use crawler thread runner

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    03ae021 View commit details
    Browse the repository at this point in the history

  14. Fix linting issue

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    9259b5c View commit details
    Browse the repository at this point in the history

  15. Define configuration options

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    c548908 View commit details
    Browse the repository at this point in the history

  16. Create initial extraction logic

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    b0475b4 View commit details
    Browse the repository at this point in the history

  17. Refactor source and scraper

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    b6e1a1a View commit details
    Browse the repository at this point in the history

  18. Cleanup scrapy_source

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    a195488 View commit details
    Browse the repository at this point in the history

  19. Adjust variable name

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    e2e6162 View commit details
    Browse the repository at this point in the history

  20. Update module exports

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    cbbf126 View commit details
    Browse the repository at this point in the history

  21. Cleanup helpers

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    c7b77eb View commit details
    Browse the repository at this point in the history

  22. Add readme contents

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    6ff6f56 View commit details
    Browse the repository at this point in the history

  23. Refactor spider vs callback precedence logic

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    014de0a View commit details
    Browse the repository at this point in the history

  24. Adjust readme

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    4fc4174 View commit details
    Browse the repository at this point in the history

  25. Add Run the pipeline placeholder

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    9a541dd View commit details
    Browse the repository at this point in the history

  26. Update Running the pipeline section

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    d427aad View commit details
    Browse the repository at this point in the history

  27. Update readme

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    86261e6 View commit details
    Browse the repository at this point in the history

  28. Add diagram

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    995c481 View commit details
    Browse the repository at this point in the history

  29. Add diagram file

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    51921fe View commit details
    Browse the repository at this point in the history

  30. Update readme

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    a28769d View commit details
    Browse the repository at this point in the history

  31. Update readme

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    2ba04bb View commit details
    Browse the repository at this point in the history

  32. Update function signature

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    654290d View commit details
    Browse the repository at this point in the history

  33. Fix mypy linting issues

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    702e4fe View commit details
    Browse the repository at this point in the history

  34. Accept start_urls from param

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    f6a3ce5 View commit details
    Browse the repository at this point in the history

  35. Use default dictionary if no config found

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    48e4b93 View commit details
    Browse the repository at this point in the history

  36. Cleanup code

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    99dcbd6 View commit details
    Browse the repository at this point in the history

  37. Fix mypy issues

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    3010970 View commit details
    Browse the repository at this point in the history

  38. Fix mypy issues

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    8640e04 View commit details
    Browse the repository at this point in the history

  39. Fix mypy issues

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    2282da8 View commit details
    Browse the repository at this point in the history

  40. Refactor source/resource scaffolding

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    e8b2ff0 View commit details
    Browse the repository at this point in the history

  41. Fix typing issue

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    1d332e2 View commit details
    Browse the repository at this point in the history

  42. Adjust scraping test

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    74663c2 View commit details
    Browse the repository at this point in the history

  43. Fix typing issue

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    a077118 View commit details
    Browse the repository at this point in the history

  44. Fix typing issue

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    19f5a4a View commit details
    Browse the repository at this point in the history

  45. Implement generic base queue for typing

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    24608ec View commit details
    Browse the repository at this point in the history

  46. Remove unused import

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    122d3f7 View commit details
    Browse the repository at this point in the history

  47. Format code

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    cb6c9ca View commit details
    Browse the repository at this point in the history

  48. Update readme

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    b588d96 View commit details
    Browse the repository at this point in the history

  49. Explicitly type next page resolver

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    397a3d5 View commit details
    Browse the repository at this point in the history

  50. Update readme

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    3d90736 View commit details
    Browse the repository at this point in the history

  51. Add pytest-mock

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    d6789d6 View commit details
    Browse the repository at this point in the history

  52. Add test to check callback calls

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    f767da6 View commit details
    Browse the repository at this point in the history

  53. Update readme

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    f70534f View commit details
    Browse the repository at this point in the history

  54. Add more tests

    @sultaniman
    sultaniman committed Jan 18, 2024
    Configuration menu
    Copy the full SHA
    97f3433 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. Adjust next page checks

    @sultaniman
    sultaniman committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    95a00fc View commit details
    Browse the repository at this point in the history