Exceptions on framework thread are not visible

[abroekhuis]

I have a testcase in which (due to a programming error) an exception was raised in a callback method called by the framework (servicetracker). This exception never showed up in the testcase, resulting in false positives. In a real-live run it did fail however.
Is there some way to catch exceptions from the fw thread? IMO any exception in a testcase should fail that test.

Thanks!

[kriegfrj]

This is a really good question. It's an interesting and challenging problem to solve. You can register a framework listener and get alerted to exceptions that way. The trick would be synchronizing with the running test thread, and also knowing if the exception is related to your test. I think we could come up with something that would work though.

[bjhargrave]

Normally when a callback thread calls code which throws an unexpected exception, the callback thread must decide how to handle the exception. It may log it. It may pass it to an uncaught exception handler. But an exception thrown in another thread will not be visible in the thread running the test case as there is no likely place for such an exception to be thrown from.

In the case of a framework callback thread, it will likely log it to the OSGi Log Service.

[tjwatson]

In the case of a framework callback thread, it will likely log it to the OSGi Log Service.

I suspect in most cases the framework will publish a FrameworkEvent error. That does end up getting published as a LogEvent if there is a log service implementation present.

[abroekhuis]

While I get that a callbackthread should handle the exception itself, for testing it would be useful (if not needed) to catch runtime exceptions, after all, it is a testing framework to find problems.

Isn't there something possible with the bnd test launcher? Catch any exceptions in there and fail on them?

Alternatively, If those exception are published as FrameworkEvents, would it be possible to have an Extension that handles them and fails a test?

[kriegfrj]

While I get that a callbackthread should handle the exception itself, for testing it would be useful (if not needed) to catch runtime exceptions, after all, it is a testing framework to find problems.

Isn't there something possible with the bnd test launcher? Catch any exceptions in there and fail on them?

Alternatively, If those exception are published as FrameworkEvents, would it be possible to have an Extension that handles them and fails a test?

Yes, that is possible. I was thinking something along those lines. One trick is that the framework events will not be delivered synchronously - the test that caused it may be finished by the time the corresponding framework event is received.

Also, if the framework hits a bump serious enough to cause an exception, there will likely be other side effects. You could test for those side effects.

I'm sure something is possible here - it will just require a bit of thought. I think @stbischof already has a draft pr for capturing and asserting on framework events - we may be able to (metaphorically) take that one out and dust it off.

[tjwatson]

If the test framework could do something to trigger a framework event post test then it could block on that event to happen (for example, call refreshPackages on a "dummy bundle" that would cause a PACKAGES_REFRESHED event). This would ensure any FrameworkEvent ERROR events got published and captured by the test framework. This way you could detect if any unexpected errors occurred during the test.

[rotty3000]

@abroekhuis could you provide a testable case that throws a framework error so we can start prototyping?

[abroekhuis]

@abroekhuis could you provide a testable case that throws a framework error so we can start prototyping?

Sure, probably will take a few days, but I will make something!

[bjhargrave]

Isn't there something possible with the bnd test launcher? Catch any exceptions in there and fail on them?

I don't think that is wise or possible. There can be many exceptions reported via framework listeners that should not result in the failure/termination of the launched runtime.

Alternatively, If those exception are published as FrameworkEvents, would it be possible to have an Extension that handles them and fails a test?

Yes, but (1) only you can know what exceptions should be considered test case failure vs other possible benign exceptions sent to framework listeners. There is also the timing in that events are delivered asynchronously to framework listeners. So any such extension would have to wait for some time in case an interesting exception may flow to framework listeners in order to correlate the exception with a specific test method. Such negative-test would add considerable time (1 sec?) to each test method. I suppose you could wait until the test class completes for wait for any interesting exceptions to flow, but then you would lose correlation to the test method which may have provoked the exception.

In general, this is a hard problem so solve in a callback based programming model. The callback caller must protect against unexpected exceptions from the callback as the callback caller has many more callbacks to call. And one ill-behaved callback must not break the system by callback caller's thread to die. OSGi Promises have this basic issue the handle it by delivering the exception to the thread's uncaught exception handler. Since the Promise user can control the threads used for callbacks, they can install an appropriate uncaught exception handler to detect these errors during testing.

[kriegfrj]

Isn't there something possible with the bnd test launcher? Catch any exceptions in there and fail on them?

I don't think that is wise or possible. There can be many exceptions reported via framework listeners that should not result in the failure/termination of the launched runtime.

Alternatively, If those exception are published as FrameworkEvents, would it be possible to have an Extension that handles them and fails a test?

Yes, but (1) only you can know what exceptions should be considered test case failure vs other possible benign exceptions sent to framework listeners. There is also the timing in that events are delivered asynchronously to framework listeners. So any such extension would have to wait for some time in case an interesting exception may flow to framework listeners in order to correlate the exception with a specific test method. Such negative-test would add considerable time (1 sec?) to each test method. I suppose you could wait until the test class completes for wait for any interesting exceptions to flow, but then you would lose correlation to the test method which may have provoked the exception.

In general, this is a hard problem so solve in a callback based programming model. The callback caller must protect against unexpected exceptions from the callback as the callback caller has many more callbacks to call. And one ill-behaved callback must not break the system by callback caller's thread to die. OSGi Promises have this basic issue the handle it by delivering the exception to the thread's uncaught exception handler. Since the Promise user can control the threads used for callbacks, they can install an appropriate uncaught exception handler to detect these errors during testing.

These are the reasons why I was thinking it would be more appropriate to have a library (something like what @stbischof prototyped) to capture and assert on framework events, If you're interested in them then you can use this to look out for the exceptions that you might be interested in; if you're not then you can avoid the overhead of synchronizing with the framework thread.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.

[github-actions]

This issue has been automatically closed due to inactivity. If you can reproduce this or if you have a good use case for this feature, please feel free to reopen the issue with steps to reproduce, a quick explanation of your use case or a high-quality pull request.