[misc] Commit read offsets in Kafka integration tests

[pnowojski]

Previously offsets were not committed so the same records could be read more then once.
It was not a big issue, because so far this methods were used only for at-least-once tests.

[tzulitai]

Shouldn't we actually be deleting the topics after the test finishes?

[pnowojski]

Yes and as far as I know, we are doing this. Why do you ask?

[tzulitai]

@pnowojski only checking that we're actually not reading the same topic again in the tests.
So, the change in this PR is just to make sure that in the case we do do that using the getAllRecordsFromTopic method, we don't re-read data for verification, correct?

[tzulitai]

Maybe some method Javadoc explaining that would be nice.
From the method name getAllRecordsFromTopic, the behaviour isn't that obvious.

[tzulitai]

I would like to step back a bit and revisit tests that use this method by first discussing:
wouldn't it be more appropriate to have a validating mapper function that throws a SuccessException once it sees all records? And instead of having some timeout in this method, we have a timeout set on the test method.
This test pattern is consistently used by the integration tests for the Kafka consumer (and many other places too).

[pnowojski]

For consumer side or mapper side it is natural to use that kind of validating mappers, because you could just add them at the end of your pipeline.

For producers tests it isn't, because you need to spawn additional Flink job for this purpose, which seems unnatural to me. It would add a test dependency to a consumer code (bug in consumer would/could brake producer tests making the error messages very confusing). Furthermore using second Flink job would be definitely more heavy and more time/resources consuming - this second job would need to execute exactly same code as those methods, but wrapped into additional layer (Flink application). Lastly this wrapping would add additional complexity that could make this tests more prone for intermittent failures and timeouts.

If you have the data written somewhere, why don't you want to read them directly? One more bonus reason for doing it as it is, it makes possible to test producers without spawning any Flink job altogether in some mini IT cases (which I'm doing in tests for Kafka011, I test FlinkKafkaProducer directly).

[tzulitai]

@pnowojski alright, that makes sense.
You don't actually need a separate Flink job because you can just add a completely non-attached graph that consumes from the topic within the same job. But I agree that it complicates the test scenario and would also have to consider bugs in the consumer.

[tzulitai]

Travis seems to have a large amount of abnormal timeouts, though. I'm not sure if it is really related to this change or otherwise. Could you do a rebase on the latest master so that the recent Travis build changes are included, and we wait for another Travis run?

[pnowojski]

I didn't know that you can have disconnected graph in Flink :)

It shouldn't be caused by this commit, since it is included in my other PR. Rebased and let's make sure that it passes.

[pnowojski]

yes, Travis passes @tzulitai :)

[tzulitai]

Ok :) LGTM, merging ..

[pnowojski]

Thanks!