Should we have automatic test compatibility with popular external plugins?

[pc223]

I think we should setup CI to make sure the compatibility popular external plugin like Everything (or maybe all of the plugins in the manifest).

if a popular plugin breaks, CI should at least warn us that a plugin break, but probably should still produce .exe releases, because it might not be Flow's fault.

Thought?

[pc223]

Can we even do something like GUI acceptance tests with Appveyor? So we can write tests for plugins as normal users use them

https://help.appveyor.com/discussions/questions/678-gui-unit-testing-in-appveyor

[taooceros]

Instead of testing plugin compatibility, I think we should have a set of unit test for our public api, which is what they depends on. Everything plugin is a special one, which includes the infrastructure dll so may cause some issue, but other standard plugin should work fine if we don't change our public api. No need for special plugin functionality test.

[JohnTheGr8]

@pc223 I disagree with the premise: things should not break a lot, especially for something as fundamental to the project as the plugin system. Minor optimizations/improvements can be made over time but not that often.

I see the usefulness of plugin integration tests when it comes to quickly checking PRs to the manifest repository, but I don't see any need to prioritize that at this stage...

@taooceros would it have been a good idea to completely strip the plugin package of everything except the interfaces, and move all that to a different package? That way the plugin package rarely ever changes... Maybe we can still do that while it's not too late?

[pc223]

I'm not saying it should break, we all want software not to break 😄 I'm saying it did and it seems will continue to break.

I think have thorough unit tests for PublicAPI is a good approach. Although I prefer users of popular plugins don't feel "risky" when upgrade to a newest release of Flow 😂 Flow should guarantees newest releases will always works with popular plugins newest versions.

[jjw24]

well let me know if otherwise, but i dont think our users feel or should feel risky in upgrading. We havent really rolled out any if at all bad releases that breaks things for our users and their plugins. Given the amount of changes in the 5 months we bottled up for the 1.8.0 release we havent had any real issues come out of it (touch wood), i would say we are doing good. Everything plugin was the only one that had major issues during loading with 1.8.0 and that was because it's a dinosaur which we also havent given much love to despite its popularity, but this was caught and resolved during development.

overall though, yes agree with lots more tests, no doubt about that :) i would also agree to start with adding more tests around plugin api before diving into GUI or integration tests that bring a whole lot of complexity.

[taooceros]

@taooceros would it have been a good idea to completely strip the plugin package of everything except the interfaces

Yes! That's what we are going to do. Quite a few new api has been added in ipublicapi in 2.0.0, which should be enough for everything plugin to remove the infrastructure dll.

[jjw24]

if a popular plugin breaks, CI should at least warn us that a plugin break, but probably should still produce .exe releases, because it might not be Flow's fault.

I would consider loading plugins as a successful test, anything beyond that like ui testing maybe a bit overkill?

[pc223]

Maybe UI testing is a bit much, but I still think we should have some kind of automatic testing for plugins that people install thru pm install

We can do a bit more than loaded test. Maybe something like the current Flow.Launcher.Test, but high level, focus on send a query to plugins and check if any thing crash or exception, check if ouput is empty, or the count of results make sense... Anything off then send a warning.

[jjw24]

I think a good start is to add more unit tests to safe guard the behaviours and functionalities, keep in mind flow is sorely lacking in tests so we definitely need to improve in this area. A lot of what you described in the above thread can be broken down into unit tests we just need to do a fair bit of refactoring