Unit and functional tests

[Safihre]

We really need unit and functional tests.
Biggest hurdle, especially for functional tests: we need a mock-NNTP server.
We could use NZBGet's NServ, but that kind of feels wrong...

Anyone that wants to help me with this?
Especially with the creation of this mock NNTP server.

Tasks and tests:

• Creation of a mock-NNTP server that can:
  • Respond to authentication requests
  • Respond to data-request with yEnc encoded data
• Test-platform that can start this server, send requests to it and wait for API-response
• Test-platform for starting of SABnzbd with specific parameters
• Test startup of SABnzbd
  • Correct binding to HTTP(S) ports
  • Response to occupied port
  • Response to missing dependencies or wrong encoding
• Implement functional tests - NNTP authentication
  • Response to correct login
  • Response to bad login
  • Response to SSL on non-SSL port
  • Response to bad certificates
• Implement functional tests - Downloading
  • Can it download a normal file from the mock server
    • NZB added as .nzb
    • NZB added inside .tar.gz, or .zip or .rar
  • Can it download a damaged file from the mock server and repair/unpack
  • Can it download a obfuscated file from the mock server
  • Verification using only RAR-verification (test expected fail and success)
  • Verification using only SFV-verification (test expected fail and success)
  • Can it download a normal file from the mock server with UTF8 chars in the RAR filenames or the filenames inside the RAR's (both RAR3 and RAR5)
  • Can it download a broken and/or obfuscated file from the mock server with UTF8 chars in the RAR filenames or the filenames inside the RAR's (both RAR3 and RAR5)
  • Can it detect password jobs during downloading, also when the jobs have UTF8 filenames (both RAR3 and RAR5)
  • Can it detect unwanted jobs during downloading, also when the jobs have UTF8 filenames (both RAR3 and RAR5)
  • Can it download and repair passworded jobs (both RAR3 and RAR5)
  • Test for duplicate detection
  • Can it handle 7zip files
  • Can it handle zip files
  • Can it handle join-files
• Implement functional tests - RSS filter handeling
• Implement functional tests - Other config options
  • Pre-queue script running
  • Post-processing script running
  • Notification script running
  • Ignore samples option
  • Download all par2 option
  • Testing of sorting (many sub-tests should be made here!)
  • Testing of scheduler (not sure yet how..)
• Other tests
  • Is category parsed correctly from NZB (from filename {{}}file.nzb or from tag inside NZB)
  • Is password parsed correctly from NZB (from filename file{{}}.nzb or from tag inside NZB)

[sanderjo]

Can it be an(other) existing NNTP-server?
Must the NNTP-server be in python code, or can it be in C or Go?

[sanderjo]

Can it be an(other) existing NNTP-server?
Must the NNTP-server be in python code, or can it be in C or Go?

[Safihre]

It can be another yes, and the language also doesn't really matter I think. We can use TravisCI and Appveyor to compile it before running the tests.

[Safihre]

It can be another yes, and the language also doesn't really matter I think. We can use TravisCI and Appveyor to compile it before running the tests.

[Hellowlol]

What wrong with using nzbgets nserv? It even states that it can be used with other clients.

[Hellowlol]

What wrong with using nzbgets nserv? It even states that it can be used with other clients.

[Safihre]

It's like using Android to test iOS 😁

[Safihre]

It's like using Android to test iOS 😁

[Hellowlol]

@Safihre Are you gonna use pytest?

[Hellowlol]

@Safihre Are you gonna use pytest?

[Safihre]

No clue, is that the easiest?

[Safihre]

No clue, is that the easiest?

[Hellowlol]

Yep. Its a lot less boilerplate.

[Hellowlol]

Yep. Its a lot less boilerplate.

[Safihre]

@sanderjo @Hellowlol (and anybody else wanting to help): I created a tasks list of things that should be included, maybe this helps to get things going ⛱

[Safihre]

@sanderjo @Hellowlol (and anybody else wanting to help): I created a tasks list of things that should be included, maybe this helps to get things going ⛱

[Hellowlol]

Maybe we can use twisted AS nntp server.

[Hellowlol]

Maybe we can use twisted AS nntp server.

[Safihre]

Sure, you have experience with it? I don't but it seems interesting.
It doesn't have to by high performance at all, just has to work 💯

[Safihre]

Sure, you have experience with it? I don't but it seems interesting.
It doesn't have to by high performance at all, just has to work 💯

[Safihre]

I have added super basic tests, it will start SABnzbd and then test for basic API and page-serving functionality.
At least that would prevent what happened with 2.3.0Alpha1 where the Config > Switches page didn't work.

Much more work to do!

[Safihre]

I have added super basic tests, it will start SABnzbd and then test for basic API and page-serving functionality.
At least that would prevent what happened with 2.3.0Alpha1 where the Config > Switches page didn't work.

Much more work to do!

[thezoggy]

any doc on how to add tests / run it / etc..

[thezoggy]

any doc on how to add tests / run it / etc..

[Safihre]

Will add docs when there's actually functional tests, the tests are embarrassingly simple right now.
But in case (shutdown SABnzbd first!):

pip install pytest pytest-xprocess
pytest

[Safihre]

Will add docs when there's actually functional tests, the tests are embarrassingly simple right now.
But in case (shutdown SABnzbd first!):

pip install pytest pytest-xprocess
pytest

[Safihre]

Working in this branch: https://github.com/sabnzbd/sabnzbd/tree/feature/pytest
If anyone wants to contribute, just PR against that 👍
Just added basic testing of adding and removing NZB's.

Thinking of just using the SABnzbd Giganews account for now to test downloading (with the username/password encoded of course) of some of @sanderjo's test NZB's.

[Safihre]

Working in this branch: https://github.com/sabnzbd/sabnzbd/tree/feature/pytest
If anyone wants to contribute, just PR against that 👍
Just added basic testing of adding and removing NZB's.

Thinking of just using the SABnzbd Giganews account for now to test downloading (with the username/password encoded of course) of some of @sanderjo's test NZB's.

[Hellowlol]

Good stuff, i got some stuff i need to finish (sabapi for one) but ill see if i cant get started this week.

[Hellowlol]

Good stuff, i got some stuff i need to finish (sabapi for one) but ill see if i cant get started this week.

[Safihre]

AppVeyor was added to also enable automatic Windows platform testing.

[Safihre]

AppVeyor was added to also enable automatic Windows platform testing.

[Hellowlol]

http://flylib.com/books/en/2.407.1.69/1/ describe how to setup a minimal nntp server using twisted. Ill try to play with it and see how it goes.

[Hellowlol]

http://flylib.com/books/en/2.407.1.69/1/ describe how to setup a minimal nntp server using twisted. Ill try to play with it and see how it goes.

[Safihre]

@Hellowlol interesting, very extensive. Doesn't include the yEnc encoding as far as I can see, but good basis!

[Safihre]

@Hellowlol interesting, very extensive. Doesn't include the yEnc encoding as far as I can see, but good basis!

[OldskoolOrion]

I will start looking into this as well as first task.. I used Mozilla's MailMan FakeServer back in the day which was really nice for functional testing of imap-, smtp-, nntp-protocol stuff etc (conforming to rfc) - lot of examples there to start with and easy integratable in automatic workflow by the looks of it. Need to dive in and get my hands dirty to be 100% sure its a good choice of course, since I'm not yet knee-deep in sab's codebase either, and have no helicopter view yet of all the needed reqs.

Yes it's old and yes it's in that horrible javashit non-language, but if it do exactly what we want and is extremely easy to extend and adept, it's a valid option I think.

Besides that I already checked out some nice Golang based nntp-server & client stuff, which is OS, well supported and even in use live, let alone just as local server for functional testing.. I'm definitely going to dive deeper into that as well, since I still have to start teaching myself Go on a more than basic-level. Very platform-independent, quite fast, good libs and friendly community with quality server-client related packages out there.

And if all this plus stuff that's under investigation fails for us.. then there's always nserv.. but I agree.. out of respect to sab itself and nzbget as well.. it would be more elegant if sab wouldn't have to rely on nserv :-)

Lotsa spare time tomorrow to start rocking.. documentation has been collected.. a lot of source code has been git'd.
Good start to getting to know the codebase while actually doing something useful :-)

[OldskoolOrion]

I will start looking into this as well as first task.. I used Mozilla's MailMan FakeServer back in the day which was really nice for functional testing of imap-, smtp-, nntp-protocol stuff etc (conforming to rfc) - lot of examples there to start with and easy integratable in automatic workflow by the looks of it. Need to dive in and get my hands dirty to be 100% sure its a good choice of course, since I'm not yet knee-deep in sab's codebase either, and have no helicopter view yet of all the needed reqs.

Yes it's old and yes it's in that horrible javashit non-language, but if it do exactly what we want and is extremely easy to extend and adept, it's a valid option I think.

Besides that I already checked out some nice Golang based nntp-server & client stuff, which is OS, well supported and even in use live, let alone just as local server for functional testing.. I'm definitely going to dive deeper into that as well, since I still have to start teaching myself Go on a more than basic-level. Very platform-independent, quite fast, good libs and friendly community with quality server-client related packages out there.

And if all this plus stuff that's under investigation fails for us.. then there's always nserv.. but I agree.. out of respect to sab itself and nzbget as well.. it would be more elegant if sab wouldn't have to rely on nserv :-)

Lotsa spare time tomorrow to start rocking.. documentation has been collected.. a lot of source code has been git'd.
Good start to getting to know the codebase while actually doing something useful :-)

[Safihre]

I am trying to add a simple unit-test, but I am failing.
How would I test for example sabnzbd.misc.calc_age from a test .py file in the tests directory executed by pytest?
Anyone that can help me out? :)

[Safihre]

I am trying to add a simple unit-test, but I am failing.
How would I test for example sabnzbd.misc.calc_age from a test .py file in the tests directory executed by pytest?
Anyone that can help me out? :)

[Hellowlol]

I normally import the stuff I need in conftest.py then in test_misc.py

from conftest import misc
def test_calc_age():
    pass

[Hellowlol]

I normally import the stuff I need in conftest.py then in test_misc.py

from conftest import misc
def test_calc_age():
    pass

[Safihre]

I am considering (due to a project at work) moving the functional tests to an actual functional-testing tool like Selenium. This way we can test actual user flow instead of just API calls.

[Safihre]

I am considering (due to a project at work) moving the functional tests to an actual functional-testing tool like Selenium. This way we can test actual user flow instead of just API calls.

[wwwehr]

@Safihre perhaps one of these?

• https://testingbot.com/open-source
• https://www.browserstack.com/automate/travisci
• https://saucelabs.com/open-source

[wwwehr]

@Safihre perhaps one of these?

• https://testingbot.com/open-source
• https://www.browserstack.com/automate/travisci
• https://saucelabs.com/open-source

[james-d-elliott]

I can probably setup a server for this if we are still looking for something, and I might work out how to build a docker image for it.

[james-d-elliott]

I can probably setup a server for this if we are still looking for something, and I might work out how to build a docker image for it.