core: fix form encoding. resolves #4346 #3061

[ngosang]

This change should fix ALL POST REQUESTS in Jackett with encoding != UTF8. That mean searches #4346, login #3061 ...

[ngosang]

@garfield69 this change is big. I need help to test it.
To fix / test #4346, you also need to remove the filters in the title field. Some of them are breaking the title.
You should also try some trackers with encoding != utf8 sending post requests.

@ALERTua could you try to download the binaries generated by this PR and test #3061 ?
Linux / mac => https://ci.appveyor.com/project/Jackett/jackett/builds/30272329/job/s9li0swlqjahpvo6/artifacts
Windows / mono => https://ci.appveyor.com/project/Jackett/jackett/builds/30272329/job/mfvh5tp1vptlv64w/artifacts

[garfield69]

To fix / test #4346, you also need to remove the filters in the title field. Some of them are breaking the title.

just edit the nnm-club config and turn off the stripRussian checkbox.

Just FYI, I'm on reduced availability while the Australian Open Tennis is on (this week and the next) so I will be a while completing the testing you requested. But it will get done, eventually ;-)

[garfield69]

@ALERTua FYI, while testing using the build from the artifacts links provided here, you may want to turn ON the disable auto-updates checkbox on the jackett dashboard, followed by apply server settings button, or you may find your test-build overwritten by the latest release when its version gets ahead ;-)

[ALERTua]

I'm on it. give me an hour or so.

[ALERTua]

• disabled autoupdate
• downloaded windows binaries zip
• stopped Jackett service
• overwritten the binaries with Jackett Version 0.12.1627.0
• started the service
• made sure the Jackett is Version 0.12.1627.0
• deleted existing Rutracker indexer
• refreshed few times (capcha doesn't always appear)
• tried adding existing russian-letter login with correct password and capcha
• no luck
  

[garfield69]

note: if the captcha does not appear that simply means that rutracker is not asking for one. so simply leave it blank and continue.

[ALERTua]

oh. thanks. would be nice if this is mentioned somewhere in the UI :)

[garfield69]

firstly I confirmed that I am using the same client that you have patched httpwebclient2
Note the on windows using the artifact in this PR the default client is httpwebclient
so I shutdown the service and started jackettconsole from a CLI with
jackettconsole --UseClient httpclient2 -l -t
and the log confirms

02-05 19:37:22 Info Starting Jackett v0.12.1627.0
02-05 19:37:22 Info Logging enabled.
02-05 19:37:22 Info Tracing enabled.
02-05 19:37:22 Debug Creating web host...
02-05 19:37:22 Debug Content root path is: C:\ProgramData\Jackett\Content
02-05 19:37:22 Debug Autofac container built
02-05 19:37:22 Info Environment version: 4.0.30319.42000 (C:\Windows\Microsoft.NET\Framework\v4.0.30319\)
02-05 19:37:22 Info OS version: Microsoft Windows NT 6.2.9200.0 (64bit OS)
02-05 19:37:22 Info Jackett variant: FullFrameworkWindows
02-05 19:37:22 Info ThreadPool MaxThreads: 1023 workerThreads, 1000 completionPortThreads
02-05 19:37:22 Info App config/log directory: C:\ProgramData\Jackett
02-05 19:37:22 Info Using HTTP Client: HttpWebClient2

Then I tested by searching for Мажор on nnm-club #4346 and
I can confirm that this patch does not appear to make any difference to the nnm-club indexer results :-(
by adding

    keywordsfilters:
      - name: strdump

to the search block I can confirm that at least to begin with, the keyword contains the correct Cyrillic set

Feb 05, 2020 07:16:15 PM | Debug | CardigannIndexer (nnm-club): strdump: Мажор

But looking at the html returned by nnm-club I see

							<fieldset class="fieldset" style="margin: 2px;">
								<legend>Название содержит&nbsp;·&nbsp;<a class="med" href="#" onclick="return get_fs_link();">Получить ссылку</a></legend>
								<div style="padding: 4px">
									&nbsp;<input class="post" id="nm" type="text" size="30" maxlength="60" name="nm" value="Мажор" />&nbsp;<label for="extended" class="tit" title="Расширенный синтаксис"><input id="extended" type="checkbox" name="extended" value="1" /></label>
								</div>
							</fieldset>

so it appears that somehow Мажор has been tuned into Мажор ?!?

normally I would be able to see in the enhanced log the http GET url used by jackett for the search, but because this is a POST, the Cardigann processor does not generate a debug message :-(
It might be useful if the Cardigann processor could generate a debug message of the http post, to confirm what it requested from nnm-club site.

given that this first test failed I did not proceed to test rutracker login.

[garfield69]

Q: is c# code for HttpWebClient2NetCore === to the useclient httpwebclient2 ?!?
maybe I should use my visual studio and copy your source into my local repo and test using my .net core build instead of using the fullframeworkwindows build in this artifact?

[garfield69]

Hah! I was right. my visual studio build runs the CoreWindows variant, and uses the HttpWebClient2NetCore client.

so disregard my previous test results and I will start again using my repo source with your source over the top.

[garfield69]

• test 1: check that nnm-club search for Мажор works
• test 2: other yaml that search using post, that use utf-8, and that I have access to
  • crazycorner
• test: 3 other yaml that search using post, that do not use utf-8, and that I have access to
  • there are none. nnm-club is the only one
• test 4: ruTracker login with regular username
• test 5: ruTracker login with Cyrillic username

LGTM ;-)

you will need to code similar routines for the other 3 clients in order to cover all builds

[ngosang]

@garfield69 I think this is ready but I didn't test other WebClients, only HttpWebClient2NetCore.

[garfield69]

I've used the Jackett windows FullFrameworkWindows with HttpWebClient to test RuTracker and nnm-club with Russian and English searches. With your HttpWebClient2NetCore tests, I think we can push this one out. Thanks.