[SPARK-13618][STREAMING][WEB-UI] Make Streaming web UI page display rate-limit lines on statistics graph - Part 1

[lw-lin]

What changes were proposed in this pull request?

This PR makes Streaming web UI display rate-limit lines in the statistics graph.

Design doc: https://docs.google.com/document/d/1kGXQEcToNglDK-AbyEJGoYX9wwODimUVf0PetoxuU5M/edit#

Part 1 (this PR):

1. adds in RateLimiter a data structure keeping history of rate limit changes, so that calculating the upper bound of how many records we can receive in a block interval is possible;
2. adds the numRecordsLimit information into the path from BlockGenerator generates a Block to the ReceivedBlockInfo;

Part 2: PR #11633
Part 3: PR #11643

How was this patch tested?

• units tests
• manually checked UI(see below)

Screenshots

without back pressure

with back pressure

[AmplabJenkins]

Can one of the admins verify this patch?

[lw-lin]

2 things I'm still working on:
- better deal with situations where numRecordsLimits are mixed with Some(...) from some batches and 'None' from the other batches;
- fix compiling and tests issues in modules such as external/kafka, etc.

The streaming module compiles and passes units tests. I'm posting this at this stage hoping to get some early feedbacks so I can improve this.

Update: they are done, so this is ready for review.

@tdas @zsxwing would you take a look when you have time please? Thanks!

[zsxwing]

Why need to display the rate-limit line when backpressure is disabled?

[zsxwing]

A suggestion: could you just add the rate limit informations to some Batch info and then store history limits in StreamingJobProgressListener so that we don't need to add history limits to RateLimiter?

[lw-lin]

@zsxwing
Yes you are right -- indeed no need to display rate-limit line when an InputDStream instance is not under rate control: I've added a field underRateControl for this purpose (please see Part 2) and updated the screenshots accordingly. Thanks !

Regarding where to collect the rate limit numbers -- at driver side or at executor side -- I've attached an design doc (https://docs.google.com/document/d/1kGXQEcToNglDK-AbyEJGoYX9wwODimUVf0PetoxuU5M/edit#), in which there is some discussion. Thanks for pointing this out !

[lw-lin]

All three parts are now ready for review. I've also drafted a design doc (https://docs.google.com/document/d/1kGXQEcToNglDK-AbyEJGoYX9wwODimUVf0PetoxuU5M/edit#), hopefully it can help reviewers review this more faster.

@tdas @zsxwing would you mind taking a look when you have time? Thanks!

[lw-lin]

This PR has many conflicts to resolve, so I'm closing this for now and will re-open later, thanks.