BOM should take precedence over Content-Type header when detecting the encoding

[mtabbasi]

closes #5601

[codecov]

Codecov Report

Merging #5611 (568b309) into master (52d9349) will decrease coverage by 0.44%.
The diff coverage is 100.00%.

❗ Current head 568b309 differs from pull request most recent head a988c4b. Consider uploading reports for the commit a988c4b to get more accurate results

@@            Coverage Diff             @@
##           master    #5611      +/-   ##
==========================================
- Coverage   88.85%   88.40%   -0.45%     
==========================================
  Files         162      162              
  Lines       10962    10850     -112     
  Branches     1894     1861      -33     
==========================================
- Hits         9740     9592     -148     
- Misses        942      975      +33     
- Partials      280      283       +3     

Impacted Files	Coverage Δ
scrapy/http/response/text.py	100.00% <100.00%> (ø)
scrapy/robotstxt.py	73.33% <0.00%> (-24.20%)	⬇️
scrapy/core/downloader/handlers/__init__.py	83.33% <0.00%> (-9.40%)	⬇️
scrapy/squeues.py	92.85% <0.00%> (-7.15%)	⬇️
scrapy/pqueues.py	95.86% <0.00%> (-3.31%)	⬇️
scrapy/extensions/memdebug.py	45.00% <0.00%> (-2.62%)	⬇️
scrapy/downloadermiddlewares/cookies.py	95.74% <0.00%> (-2.16%)	⬇️
scrapy/extensions/debug.py	44.73% <0.00%> (-1.42%)	⬇️
scrapy/extensions/throttle.py	44.89% <0.00%> (-1.11%)	⬇️
scrapy/core/downloader/handlers/http11.py	93.06% <0.00%> (-1.05%)	⬇️
... and 61 more

[Gallaecio]

Looks good to me. Could you add a test for it?

[mtabbasi]

Sure

[Gallaecio]

Nice work!

[kmike]

Thanks @mtabbasi! I like how clean is this PR.