Run tests against wordpress.org plugin directory

[crazytonyli]

Note

This PR is built on top of #415

This PR updates plugin directory tests to call api.wordpress.org directly instead of using cached content.

Since there are 44k+ plugins on wordpress.org and the whole tests take 90 minutes, the full tests run on nightly build by default. In regular CI builds and local development, the tests only verify a small amount (20) of plugins. You can set TEST_ALL_PLUGINS env to run full tests locally.

A whole bunch of tests that verifies weird JSON response values are removed in this PR, but they will be added back later in a separate PR. Update: I have added them in this PR.

[jkmassel]

IMHO we shouldn't hit the endpoint at all if we're not running a full test.

I see you have a json file for testing, can we have one of those for this so that it runs quickly? We're just interested in testing the parsing code, so I'd love to factor out networking for the vast majority of cases if we can

[crazytonyli]

IMHO we shouldn't hit the endpoint at all if we're not running a full test.

Considering this test is part of integration tests, I think it's okay for it to make a small amount of HTTP requests? There is still value in running this test during PR (and locally development), like running the entire test function to ensure there is not obvious issue in the test code itself.

I see you have a json file for testing, can we have one of those for this so that it runs quickly?
I have added one in 022fa1f as part of unit tests.

[oguzkocer]

@crazytonyli I'll try to review this one tomorrow. Sorry for the delay!

[oguzkocer]

I left a few minor suggestions, one of which is enabling a test that was missing the #[test] attribute. As it turns out, that test is currently failing.

I also tested the integration tests - without the TEST_ALL_PLUGINS environment variable - and even though the test reports that it was successful, there were actually parsing errors:

Only a small amount of plugins will be fetched.
F(https://api.wordpress.org/plugins/info/1.2/?action=query_plugins&request[per_page]=10&request[page]=1)F(https://api.wordpress.org/plugins/info/1.2/?action=query_plugins&request[per_page]=10&request[page]=2)
Fetching and parsing 0 plugins...

2 query plugins failures:

• "https://api.wordpress.org/plugins/info/1.2/?action=query_plugins&request[per_page]=10&request[page]=1" : reqwest::Error { kind: Decode, source: Error("missing field preview_link", line: 1, column: 28122) }
• "https://api.wordpress.org/plugins/info/1.2/?action=query_plugins&request[per_page]=10&request[page]=2" : reqwest::Error { kind: Decode, source: Error("missing field preview_link", line: 1, column: 8573) }
  0 plugin information failures:
  test test_parsing_full_plugin_directory ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.59s

We should report the test as failure in such a case.

---

I really don't like using environment variables for configuration. In general, I feel like environment variables are very convenient, but they can also cause a headache. There are many places it can be set from and they are hard to track. Furthermore, the behavior change in the code can feel magical, because it's not clear why it changed unless you know the existence of an environment variable - meaning it lacks intentionality.

I've been very deliberate in avoiding environment variables until now and I'd strongly prefer us continue to do so. We can merge this PR with the environment variable, but I'd like to try and open a follow up PR to replace it. Let me know if you have any concerns about that plan.

[crazytonyli]

@oguzkocer Good catch. I have fixed all the issues.

Let me know if you have any concerns about that plan.

Not at all.