the --source option.

[jefftriplett]

The "A First Scraper" walk-through was pretty good, and using it as a guide, I built a scrapper and chained a few pages together.

One point of confusion for me was how to use the scraper once I was done. Using the test subcommand and --source option made a lot of sense, and I quickly was iterating using the scrape subcommand with the --fastmode option. I assumed once I was done that I could use the --source option with scrape but then realized that it wasn't supported.

Is this a design goal? Without it, I'm not as sure how to make this reusable unless it's assumed that I embed spatula inside a Python app. That's fine too. I just couldn't quite get there yet from the docs.

---

Update:

I figured out how to bootstrap a starting Page which worked well for me. Sharing here in case it's useful for someone else:

class ExamplePage(Page):
    source = NullSource()

    def process_page(self):
        yield PageDetail(source=f"https://example.com/1")
        yield PageDetail(source=f"https://example.com/2")
        yield PageDetail(source=f"https://example.com/3")
        yield PageDetail(source=f"https://example.com/4")
        yield PageDetail(source=f"https://example.com/5")
        yield PageDetail(source=f"https://example.com/6")

[jamesturk]

I hadn't thought about use cases where the --source option would be useful on the other commands, but I see no reason why it shouldn't be available for others so I've added it in 0.8.1.

In the cases where I've needed multiple pages I have used a NullSource too. Great reminder to document that as I get into documenting more of the idioms I've found useful.

[jamesturk]

& by the way, thanks so much for kicking the tires on this & giving it a try!

[jefftriplett]

No problem. It was fun to add a scraper for a project I have been putting off for years. This feels much easier to test/troubleshoot too.

[jefftriplett]

I am new to discussions, but I'm going to close this. I should have probably added it to another category. Sorry about that :)