'=' added at end of urls extracted from LinkExtractor

[gmargari]

$ scrapy shell "https://uat.payleap.com/transactservices.svc"
> from scrapy.linkextractors import LinkExtractor
> [ l.url for l in LinkExtractor(allow=()).extract_links(response) ]
['https://uat.payleap.com/TransactServices.svc?wsdl=']

Scrapy shell info:

2016-05-11 16:37:21 [scrapy] INFO: Scrapy 1.0.5 started (bot: scrapybot)
2016-05-11 16:37:21 [scrapy] INFO: Optional features available: ssl, http11
2016-05-11 16:37:21 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0, 'DUPEFILTER_CLASS': 'scrapy.dupefilters.BaseDupeFilter'}
2016-05-11 16:37:22 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-05-11 16:37:22 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-05-11 16:37:22 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-05-11 16:37:22 [scrapy] INFO: Enabled item pipelines: 
2016-05-11 16:37:22 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6024
2016-05-11 16:37:22 [scrapy] INFO: Spider opened
2016-05-11 16:37:23 [scrapy] DEBUG: Crawled (200) <GET https://uat.payleap.com/transactservices.svc> (referer: None)

[redapple]

@gmargari , this is the (unfortunate) behavior of canonicalize_url (enabled by default in LinkExtractor).

You can disable it though:

In [1]: from scrapy.linkextractors import LinkExtractor

In [2]: [ l.url for l in LinkExtractor(allow=()).extract_links(response) ]
Out[2]: ['https://uat.payleap.com/TransactServices.svc?wsdl=']

In [3]: [ l.url for l in LinkExtractor(allow=(), canonicalize=False).extract_links(response) ]
Out[3]: ['https://uat.payleap.com/TransactServices.svc?wsdl']

[gmargari]

Thanks!

[Digenis]

@gmargari, see #1941