[BEAM-7726] Implement State Backed Iterables in Go SDK

[lostluck]

This PR does a few things towards adding State Backed Iterables

• Adds unit tests for the existing iterable protocols.
• Refactors the two disparate read loops in datasource.go into a single loop, and introduces helper methods to clean up repeated code.
  • This further helps prepare for [BEAM-490] where CoGBK becomes first class.
• Renames code that is fairly State API centric to to State from SideInput.
• Most of the code was to help use the StateAPI for Side inputs, but it was really rather general.
• Note: This only helps with Reading, and in particular Runner iterables from the state API, not doing anything towards Writing.
• Actually implement the State backed protocol (with unit test)

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[lostluck]

R: @robertwb, @lukecwik
Note: This code diverges from the stated protocol for Iterables in beam_runner_api.proto [1] where multi-chunk iterables are prefixed with a 0. As implemented Go SDK, and Python SDKs [2] currently require a -1 instead. Either the SDKs need to change or the proto documentation should change to match.

[1] https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L582
[2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/coders/coder_impl.py#L702

[lostluck]

Run Go PreCommit

[lostluck]

cc: @herohde In case you've got thoughts on the refactoring.

Otherwise, I'm investigating the PreCommit failure. It looks like it's a Go Gradle thing WRT vendoring. It might be time to do away with it.

[lostluck]

cc: @ananvay

[lostluck]

R: @youngoli

[lostluck]

Run Go PreCommit

[youngoli]

I have two small clarification questions but no actual issues, so LGTM (edit: once it passes Go Precommit of course).

Had a little trouble understanding the implementation of datasource_test.go, so hopefully there's no errors hidden in the implementation. But test coverage-wise it looks good.

[robertwb]

R: @robertwb, @lukecwik
Note: This code diverges from the stated protocol for Iterables in beam_runner_api.proto [1] where multi-chunk iterables are prefixed with a 0. As implemented Go SDK, and Python SDKs [2] currently require a -1 instead. Either the SDKs need to change or the proto documentation should change to match.

[1] https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L582
[2] https://github.com/apache/beam/blob/master/sdks/python/apache_beam/coders/coder_impl.py#L702

Ah, thanks for catching that. The docs are wrong (java uses -1 as well [3], and otherwise there'd be ambiguity in encoding an empty list). #9099

[3] https://github.com/apache/beam/blob/release-2.13.0/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/IterableLikeCoder.java#L101

[lostluck]

Run Go PostCommit

[lostluck]

It seems that between Go Gradle, and the version of go we have it run (go 1.10) it looks like the Beam Go SDK is no longer compatible with go 1.10.
In a separate client, I validated that the gradle build is much happier with go 1.12 (which gogradle manages by itself anyway). I'll resolve that first in a separate PR and then ensure this passes here before merging it in.

[lostluck]

Run Go PreCommit

[lostluck]

Run Go PostCommit