Jan 15, 2019
Merge #33718
33718: release-2.1: build,Makefile: break up the release process into two phases r=bobvawter a=danhhz

Backport 2/2 commits from #33516.

/cc @cockroachdb/release

---

I need the bound the time I can spend babysitting the next release if
anything goes wrong, so this is done in a purely additive manner which,
in the worst case, will allow falling back to the previous release
process. Once we're happy with the new one, we can clean up the old code
and scripts.

The first phase is to build provisional binaries. These are exactly the
bits we intend to ship as whatever version we're releasing. The
provisional SHA is marked in git by a tag with the form
`provisional_yyyymmddhhss_<semver>`. The various binaries (a combination
of platform, fully static vs not, race vs not) are uploaded to their
final places in s3, but nothing points at them.

At this point, the release coordinator will stage these binaries on the
long running clusters for verification. A followup PR will also allow
them to be used for a run of the nightly and weekly roachtests.

If any issues are found, a new SHA is selected, a new provisional tag is
pushed for it, binaries are built again, and verification is resumed.

Once we have provisional binaries that we're happy with, they're
blessed, which is the second phase. This makes and uploads the source
archive artifact. If the release is the latest stable release, it also
updates the various latest pointers.

The continuous publication of binaries for every green master build is
still supported but currently a bit of a hack. Ideally, the teamcity
build for it would work by releasing the provisional binaries and
immediately blessing them. However, the latest pointers work differently
between the two, so for now, this entire process is left inside the
provisional binaries step and blessing is disallowed for non-releases.
Perhaps we can unify the behaviors a bit and clean this up at some
point. The standalone workload binary is no longer continuously
published, since it's now built into the cockroach binary.

To pull all of this off, a small bit of functionality had to be added to
the Makefile. The version that a binary self reports is usually
determined automatically based on the "closest" git tag of whatever
checkout built it. We don't want the provisional tags to be considered
in this, so when we build the provisional binary, we need to override
the version. A BUILDINFO_TAG override is added for this to the various
ways to make a binary `make {mkrelease,build,...}` as well as `make
archive`.

As far as I can tell, everything is uploaded to the same place and with
the same folder and binary names as before, _except_ the folder inside
the artifact for the latest binary is now `cockroach_<semver>_<suffix`
instead of `cockroach_latest_<suffix>`. This should be fine, since both
`crl-stage-binaries` and `roachprod stage untar it with
`--strip-components 1`. (In fact, I think finally doing the TODO to
always make latest a redirect may be as simple as adding the `-L` flags
to curl in those two places.)

Release note: None


Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
Jan 11, 2019
Revert "storage: allow HeartbeatTxn and EndTxn requests to create txn…
… records"

This reverts commit 8143b45.

Release note: None
Jan 11, 2019
Revert "storage: allow HeartbeatTxn and EndTxn requests to create txn…
… records"

This reverts commit 8143b45.

Release note: None
Jan 8, 2019
Merge #33516
33516: build,Makefile: break up the release process into two phases r=bdarnell a=danhhz

I need the bound the time I can spend babysitting the next release if
anything goes wrong, so this is done in a purely additive manner which,
in the worst case, will allow falling back to the previous release
process. Once we're happy with the new one, we can clean up the old code
and scripts.

The first phase is to build provisional binaries. These are exactly the
bits we intend to ship as whatever version we're releasing. The
provisional SHA is marked in git by a tag with the form
`provisional_yyyymmddhhss_<semver>`. The various binaries (a combination
of platform, fully static vs not, race vs not) are uploaded to their
final places in s3, but nothing points at them.

At this point, the release coordinator will stage these binaries on the
long running clusters for verification. A followup PR will also allow
them to be used for a run of the nightly and weekly roachtests.

If any issues are found, a new SHA is selected, a new provisional tag is
pushed for it, binaries are built again, and verification is resumed.

Once we have provisional binaries that we're happy with, they're
blessed, which is the second phase. This makes and uploads the source
archive artifact. If the release is the latest stable release, it also
updates the various latest pointers.

The continuous publication of binaries for every green master build is
still supported but currently a bit of a hack. Ideally, the teamcity
build for it would work by releasing the provisional binaries and
immediately blessing them. However, the latest pointers work differently
between the two, so for now, this entire process is left inside the
provisional binaries step and blessing is disallowed for non-releases.
Perhaps we can unify the behaviors a bit and clean this up at some
point. The standalone workload binary is no longer continuously
published, since it's now built into the cockroach binary.

To pull all of this off, a small bit of functionality had to be added to
the Makefile. The version that a binary self reports is usually
determined automatically based on the "closest" git tag of whatever
checkout built it. We don't want the provisional tags to be considered
in this, so when we build the provisional binary, we need to override
the version. A BUILDINFO_TAG override is added for this to the various
ways to make a binary `make {mkrelease,build,...}` as well as `make
archive`.

As far as I can tell, everything is uploaded to the same place and with
the same folder and binary names as before, _except_ the folder inside
the artifact for the latest binary is now `cockroach_<semver>_<suffix`
instead of `cockroach_latest_<suffix>`. This should be fine, since both
`crl-stage-binaries` and `roachprod stage untar it with
`--strip-components 1`. (In fact, I think finally doing the TODO to
always make latest a redirect may be as simple as adding the `-L` flags
to curl in those two places.)

Release note: None

Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
Dec 12, 2018
Merge pull request #33065 from petermattis/backport2.1-33063
release-2.1: c-deps: fix an infinite loop in MVCCScan
Dec 10, 2018
Merge #32774
32774: opt: Add Update support to CBO r=andy-kimball a=andy-kimball

Add a new Update operator to the CBO which is used to optimize the
UPDATE SQL statement. As part of compiling the UPDATE statement, the
optbuilder constructs an expression that outputs the existing values
for all rows from the target table that match the WHERE clause.
Additional column(s) that provide updated values are projected for
each of the SET expressions, as well as for any computed columns. For
example:
```
  CREATE TABLE abc (a INT PRIMARY KEY, b INT, c INT)
  UPDATE abc SET b=1 WHERE a=2
```
This would create an input expression similar to this SQL:
```
  SELECT a AS oa, b AS ob, c AS oc, 1 AS nb FROM abc WHERE a=2
```
The execution engine evaluates this relational expression and uses
the resulting values to form the KV keys and values.

Co-authored-by: Andrew Kimball <andyk@cockroachlabs.com>
Dec 5, 2018
Merge pull request #32853 from nvanbenschoten/backport2.1-32773
release-2.1: storage: don't clear lastTxnMeta on WriteIntentError to different key
Nov 28, 2018
Merge pull request #32507 from knz/20181120-fix-32505
sql: catch invalid functions in the UPDATE SET RHS (v2.0.x)
Nov 14, 2018
Merge pull request #32318 from bdarnell/backport-disable-timebound
backport-2.1: batcheval: Disable time-bound iterators in KV commands
Nov 12, 2018
Merge #32211
32211: batcheval: Work around time-bound iterator bug in RefreshRange r=benesch a=bdarnell

Updates #28358
Fixes #31823

Release note (bug fix): Fix long stalls that can occur in contended
transactions.

Co-authored-by: Ben Darnell <ben@bendarnell.com>