Notebook on web crawling

[WilliamEspegren]

Why are these changes needed?

Lacking web crawling examples in both notebooks and the docs/Example.md

Checks

Not any applicable right?

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

I referred to a previous pull request for inspiration while creating this notebook example for Spider. This PR provided a helpful reference for structure and content, which I have adapted to fit the specific requirements of Spider.

Twitter handle: @WilliamEspegren

[WilliamEspegren]

@microsoft-github-policy-service agree

[sonichi]

Thanks. Would you like the notebook to be rendered on the website? If so, https://github.com/microsoft/autogen/blob/main/notebook/contributing.md#how-to-get-a-notebook-displayed-on-the-website is the guide.

[WilliamEspegren]

Thanks. Would you like the notebook to be rendered on the website? If so, https://github.com/microsoft/autogen/blob/main/notebook/contributing.md#how-to-get-a-notebook-displayed-on-the-website is the guide.

Yes please! I will look into this

[WilliamEspegren]

@sonichi This notebook will be rendered on the website right?

[sonichi]

@sonichi This notebook will be rendered on the website right?

Some metadata needs to be added for it to render. It's OK to do it in a separate PR if you like.

[WilliamEspegren]

@sonichi Sorry for that, now it is formatted correctly and passes the checks

[WilliamEspegren]

@sonichi Sorry for that, now it is formatted correctly and passes the checks

Ready to merge :)

[WilliamEspegren]

@sonichi Ready to merge :)

[WilliamEspegren]

@sonichi Still ready to merge 💯

[WilliamEspegren]

@sonichi Should I change this PR in any way?

[ekzhu]

Made some small edits.

[WilliamEspegren]

thank you @ekzhu for the review, ready to merge now :)

[rysweet]

@ekzhu - looks like OP updated this one - can you check out the docs file conflict?

[ekzhu]

@ekzhu - looks like OP updated this one - can you check out the docs file conflict?

Left a comment on an unaddressed point

[WilliamEspegren]

@ekzhu sorry, fixed now :)

[ekzhu]

Merging this for now but be aware of the upcoming new extension policy for v0.4 (currently preview) onward.