Try using stream harness support for unit testing

[Schmed]

Ken wrote:

There are classes for testing found in flink-test-utls-junit (I think this also requires mockito-all, e.g. version 1.10.19)

e.g. xxxInputStreamOperatorTestHarness, for example KeyedOneInputStreamOperatorTestHarness

Lets you test operators, including stateful operators/watermarks/etc.

There's apparently a bug in the checkpoint state management for DomainDBFunction. It would be cool to use see if we could design a test case that reproduces the problem and verifies the fix.

[Schmed]

I was able to create a DomainDBFunctionTest that seems to demonstrate DomainDBFunction's inability to properly redistribute the saved state after the parallelism is changed. I've pushed the results to my 133_stream-test-harness branch (into which I've also merged recent changes from master).

I could try fixing the bug next Wednesday, but @kkrugler should feel free to take a crack at it before then if he so chooses.

[kkrugler]

@Schmed - I've fixed the bug, and done a bit of cleanup. Changes are pushed.

But when I run mvn clean package from the command line, the DomainRankFunctionTest.test() hangs (stuck waiting for termination, log has repeated Sending heartbeat to JobManager entries).

Same thing if I run it from Eclipse. I'm thinking it might be related to switching to Flink 1.5-SNAPSHOT.

[kkrugler]

@Schmed - I think once Vivek has merged the 1.5 changes into master, you could merge/validate and close this issue, yes?

[kkrugler]

@Schmed - it would be great to also add a test for Restore pre-fetching state during restore of UrlDBFunction state.

[kkrugler]

@Schmed - It would also be great to add a test for Sync up the total active urls value with state in UrlDbFunction.

[Schmed]

I added a testRestorePreFetchingState, but it looks like you need to mock up some kind of reporter in order to gain access to the gauges created by a unit test, which seems like a lot of work.

See https://stackoverflow.com/questions/51675597/how-to-unitest-gauge-metrics-in-flink for an example.

[Schmed]

I added a testRestorePreFetchingState, but it looks like you need to mock up some kind of reporter in order to gain access to the gauges created by a unit test, which seems like a lot of work.

See https://stackoverflow.com/questions/51675597/how-to-unitest-gauge-metrics-in-flink for an example.

see also Verify metrics using reporter approach