Add conclusive tests for new code

[janakrajchadha]

While simple test cases are a good place to start, they often do not cover the complexities of real world data and examples. As I have observed while working with data from different websites, testing on real world examples leads to a better and generalized solution.
We can't cover or predict all cases we'll face beforehand but better tests ensure that the no. of cases where our code doesn't work properly is reduced by a considerable amount.
Open for suggestions on how to tackle this.

[danielballan]

I think we should build up two collection of test cases:

1. Short, contrived examples aimed to exercise specific situations
2. Real examples encountered on the web

In #108, I propose storing (1) in the git repo itself. I propose storing only the UUIDs for (2), and fetching the content from web-monitoring-db as needed when tests are executed. Some considerations:

• The content for (2) will be relatively large, and it won't be something we edit over time, so it is not a good fit for version control. It's better to fetch it.
• The content for (1) will be small, and we will improve it over time, so it's better fit for version control. By storing it in separate HTML files bundled as "package data" (as opposed to in-lined into a Python script) it's easy for contributors to experiment with it without getting up to speed on pytest.

[danielballan]

As #108 has been merged, the strategy outlined above is now in place, so I will close this question.