Fixed default value of attrs argument in SgmlLinkExtractor to be tuple

[ananana]

• Fixed bug in issue #652 (default value of attrs is now tuple)
• Wrapped tags and attrs in arg_to_iter(), so tag_func and attr_func would accept both strings and lists. (so this version of the constructor is compatible with the old one)

[kmike]

Hey @ananana,

Thanks for this PR! Could you please add some tests for it?

I like how you used arg_to_iter because it makes the interface more consistent - other parameters for SgmlLinkExtractor already use it. But as it is a new feature (strings or None were not handled properly previously) it should be mentioned in docs for SgmlLinkExtractor. Also, there is deny_extensions that doesn't use arg_to_iter - maybe fix it as well?

[ananana]

I made the following changes:

• Used arg_to_iter for deny_extensions as well, as suggested (so now it accepts strings as well)
• Added tests for attrs and for deny_extensions
• Changed docs for SgmlLinkExtractor to reflect these changes

Still left to do:

• Add tests for tags parameter

[ananana]

Added tests for tags as well. Should be ready if everything looks ok.

[nyov]

plenty good testcases to support this change, for me. Thanks very much.

+1 to merge this

(on the discussion about SGMLParser not parsing <area /> correctly, I'll add that self-closing elements are not strictly valid SGML, but XML, so this could be seen as not a bug in sgmllib. This should be a 'fix' in scrapy, if deemed necessary to fix SgmlLinkExtractor)

[dangra]

no need to create a set from a list comprehension, I know it was there but it is a good moment to replace it by

set('.' + e for e in arg_to_iter(deny_extensions))

[kmike]

as Python 2.6 is no longer supported: {'.' + e for e in arg_to_iter(deny_extensions)}

[dangra]

+1

[dangra]

LGTM. thanks!

[kmike]

I suspect that if you remove arg_to_iter and make attrs=('href') again then new tests will still pass because 'href' in 'href' is True. We may e.g. add an element with "ref" attribute to the testing suite to check that condition, but I'm fine with merging without this fix.

[ananana]

That's true, I just added a new test for that (with a "ref" attribute) for the sake of thoroughness.
There are also in the test suite similar tests for the analogous situation for the tags parameter, set to "a" then to "area" (the results for these 2 shouldn't overlap), which seemed like a more natural test case to me.

[dangra]

as last request before merging, can we get this changes squashed into a single commit?

[ananana]

Rebased and squashed.