Summary: Changes were generated by the formatter, except to fix a syntax error.

Test Plan: Existing tests.

Reviewers: jcharumilind, pgopal

Reviewed By: jcharumilind

Subscribers: smahadevan

Maniphest Tasks: T29275

Differential Revision: https://phabricator.grailbio.com/D39547

fbshipit-source-id: 949fd43

Summary:
Exposes the ETag, which is a piece of data returned by S3 that contains
a hash of information specific to a given version of a resource, outside
of the s3file package.

Test Plan: Tested in stacked diff D39535.

Reviewers: rhuang, smahadevan

Reviewed By: smahadevan

Subscribers: smahadevan

Maniphest Tasks: T29873

Differential Revision: https://phabricator.grailbio.com/D39665

fbshipit-source-id: 8078b9e

Summary:
We currently have a hard-coded duration before expiration in which
credentials are not refreshed. Making this configurable via a flag
helps some users who have `grail-access` in their .bashrc, etc.

Test Plan: Manual testing.

Reviewers: aeiser, afields

Reviewed By: aeiser, afields

Subscribers: smahadevan, afields

Differential Revision: https://phabricator.grailbio.com/D39733

fbshipit-source-id: 38ed4f3

…count

Summary:
T7463 switched the ticket-server to use HasMember API call that does lookups for neseted inheritence.

T15539 switched the ticket-server to use cross account lookups, so a user with a @grailbio.com email could be in a group from summitstudies.co.uk and still be validated as a member.

T29527 noticed that "sometimes" due to a Google API delays - the existing cross account lookup is slow.

This diff reverts the cross account lookup to use the older group.get(member) API call which should avoide the API issues, and is a simplification over the old code.

Test Plan: Verify in SBX staging, and create a test cross group lookup.

Reviewers: #shared_infrastructure, trooney

Reviewed By: trooney

Subscribers: sbagaria, joshnewman, rhuang, smahadevan, mfoster

Maniphest Tasks: T29527

Differential Revision: https://phabricator.grailbio.com/D39771

fbshipit-source-id: 335e2a8

Summary:
blacklist must be a tsv containing columns: chr, pos, blacklist

When using the --blacklist parameter, drop snps if their blacklist
column contains TRUE.

Ref T24754

Test Plan: Added unit test

Reviewers: edamato, psingh

Reviewed By: psingh

Subscribers: smahadevan

Maniphest Tasks: T24698, T24754

Differential Revision: https://phabricator.grailbio.com/D40257

fbshipit-source-id: 24b7f69

Summary:
Upgrade AWS Go SDK version to v1.23.22 (primarily
to make OSS and Grail Reflow versions compatible with each other)

- Fix tests and code which otherwise breaks in the new version, specifically in the following files:
`go/src/github.com/grailbio/base/file/s3file/s3file.go`
`go/src/github.com/grailbio/reflow/blob/s3blob/s3blob.go`
`go/src/github.com/grailbio/reflow/blob/s3blob/s3blob_test.go`
- Regenerated mocks under SCRMS

Test Plan: Unit tests

Reviewers: pgopal, O3 third-party, O15 scrms, psteed, bbentson

Reviewed By: pgopal, O3 third-party, O15 scrms, psteed, bbentson

Subscribers: jcharumilind, dnicolaou, sbagaria

Differential Revision: https://phabricator.grailbio.com/D40307

fbshipit-source-id: 596089e

…mentations

Summary:
Refactor the convenience methods in admit package in the following way:
- `admit.Do` takes a `func () (bool, error)` where the `bool` is the direct feedback to the policy's `Release()` method.
- `admit.Retry` takes a `func () (CapacityStatus, error)` where the `CapacityStatus` is one of `{Within, OverNeedRetry, OverNoRetry}` and gives feedback to both the policy's `Release()` method and on how the `admit.Retry` method should behave w.r.t retries.

Test Plan: Unit tests.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: pgopal, dnicolaou

Tags: PHID-PROJ-dtbui7u2vg2f25wzkosc

Maniphest Tasks: T30447

Differential Revision: https://phabricator.grailbio.com/D40009

fbshipit-source-id: d8a3170

Summary:
Implement an AIMD-based controller as described in:
https://docs.google.com/document/d/1Nl3UyQXTRusXDu8tIt_s9N6JxXu1vhuKeBYssyaKcGU

Test Plan: Unit tests.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: pgopal, dnicolaou

Tags: PHID-PROJ-dtbui7u2vg2f25wzkosc

Maniphest Tasks: T30447

Differential Revision: https://phabricator.grailbio.com/D40054

fbshipit-source-id: 2aee1ca

Summary:
Instead of using the AWS SDK method `IsErrorRetryable` for AWS errors
use it for all errors and fix the only test case which should be using
AWS error types anyway.

Test Plan: Unit tests

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: joshnewman

Maniphest Tasks: T30700

Differential Revision: https://phabricator.grailbio.com/D40391

fbshipit-source-id: d9440ee

Summary:
Since the move to AWS Go SDK v1.23.22, many of S3 transient errors
are being considered as non-retryable which has been causing problems
for users of bigslice particularly.

This change makes us consider all AWS errors of type `RequestError`,
regardless of underlying cause, as retryable, except in the case of context errors.

Test Plan: Unit tests

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: joshnewman

Maniphest Tasks: T30700, T30832

Differential Revision: https://phabricator.grailbio.com/D40560

fbshipit-source-id: b9933f9

Summary:
Uses fatbin's mechanism for appending zip files to the end of executables.
It's an alternative to [go_embed_data](https://github.com/bazelbuild/rules_go/blob/3762b89ad8b1d71007a4a07b194a48d505613c15/go/extras.rst#id3), which we've used (D39858), but this is
faster: go_embed_data took ~1-2 minutes to embed the deepmm binary package
(Pytorch is large) in a Go binary (classifier-evaluation-deepmm) whereas
embedbin only takes ~10 s, making the development and testing experience
much nicer. Perhaps more importantly, go_embed_data generates a ~3 GiB .go
source file, which causes things like Bazel coverage measurement to OOM,
whereas embedbin does no such thing.

I prototyped reusing fatbin's internals (instead of copying), but I ended
up copying because 1) they're not too complicated, 2) I think it's a good idea
for embedbin binaries to use different magic because they're not interchangeable
with fatbin binaries, 3) code and API reuse isn't too significant given that
fatbins unify handling of the base executable and alternatives, whereas those
are conceptually distinct in embedbin.

Test Plan:
Unit test suite adapted from fatbin's, as well as an end-to-end Bazel test that
runs the gofat tool to embed files, then checks that the files are correctly
embedded. This coverage would probably be useful in the Github version, but
I don't know how to run it since the Github version does not use Bazel, so for
now I've added this end-to-end test internally.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: jschellenberger, aeiser, sbagaria, bbentson, skrishnamurthy, mmaher, smahadevan

Maniphest Tasks: T30452

Differential Revision: https://phabricator.grailbio.com/D40476

fbshipit-source-id: 69b2103

Add the `eventlog` package which provides semi-structured event logging for convenient analytics.  Example usage:
```
sess := session.NewSession()
cw := cloudwatchlogs.New(sess)
e := NewCloudWatchEventer(cw, "myLogGroup", "myLogStream")
e.Event("rpcRetry", "org", "GRAIL", "retry", 0, "maxRetry", 10)
e.Event("machineStopped", "addr", "192.168.1.1", "duration", 3600.0, "startTime": 1584140534)
```

These events will then be available in CloudWatch Logs (and therefore easily made available in Athena) for analysis.

Summary: Allow grail-ticket to exec a command with the aws environment flags

Test Plan: n/a

Reviewers: #shared_infrastructure, joshnewman, aeiser

Reviewed By: #shared_infrastructure, aeiser

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D41635

fbshipit-source-id: 087dcd2

…ilbio#16)

See grailbio#14 (comment) for context.

Co-authored-by: Alex Wissmann <awissmann@grailbio.com>

Summary:
I'd like to make some changes to grail-access. I've wondered for a long time why
it deletes an existing Vanadium principal before it starts an authentication
flow instead of just updating the blessing. I partly think it's inelegant, but
I've also seen multiple problems in #eng-help that were caused or exacerbated
by running grail-access, canceling it, and not realizing that destroys the
credentials.

After a bit of code inspection recently, I think grail-access has quite a bit
of historical cruft in its code (handling of the principal and its blessings is
disorganized and redundant), and I think the patches to work around agent errors
could be fixed in a more principled way.

However, since we use it a lot and I'm learning as I go along, my first step is
to introduce some tests. These all invoke grail-access as if from a shell and
check its behavior and output to ensure user-visible behavior is unchanged.

The connection of this to my actual work is rather distant: I am hoping this
fixes a grail-fuse issue which will make it easier to do non-Go analysis work
and otherwise streamline some dev environment tasks and then I'll be so much
more productive (ha). I think it's worth giving grail-access some attention,
since it is almost in its initial prototype state from three years ago, but it
might just take some time, and I understand if it takes time to review.

Test Plan:
Tests the EC2 blesser. I don't know how to reasonably fake the Google oauth2
server, but that's also the more reasonable flow to test manually, so we may
need to just rely on that.

Tests pass with both Bazel and `go test`.

Reviewers: aeiser

Reviewed By: aeiser

Subscribers: smahadevan, krestivo, afields, sbagaria

Differential Revision: https://phabricator.grailbio.com/D44965

fbshipit-source-id: e2ff93d

Summary:
Mitch pointed me to some helpful references and I assembled a test for part of
the Google authentication flow. I'm not bothering with anything browser-y,
instead assuming the user used their browser to get a code (which is then
ignored). While this test skips some of the flow, I think it's still useful,
at least exercising the Vanadium bits.

Test Plan: Yes.

Reviewers: aeiser, mskinner

Reviewed By: mskinner

Subscribers: jcharumilind, smahadevan

Differential Revision: https://phabricator.grailbio.com/D45052

fbshipit-source-id: bc48505

Summary:
Allow the ticket-server email and group regex to have special characets like `-`, `_`,`+`, and `.`.

NOTE: this is not a complete regex of all supported email characters. e.g. Uppercase is not included since Google lowercases all addresses.

Closes T36541

Test Plan: Verified as part of D45302

Reviewers: #shared_infrastructure, ender, bbentson

Reviewed By: #shared_infrastructure, bbentson

Subscribers: smahadevan

Maniphest Tasks: T36541

Differential Revision: https://phabricator.grailbio.com/D45350

fbshipit-source-id: 2047926

Summary:
Add support to the ticket-server to sign SSH Certificates for easier access to systems that are managed by teams.
Add new grail-ssh command to simplify the use of the SSH Certificates.
Design doc: https://docs.google.com/document/d/1_pKjirMinqm8UykFqEm5bqAqnwLOsV3qazio0pjE80I/edit

Test Plan: Tested locally and in staging

Reviewers: treaster, #shared_infrastructure

Reviewed By: treaster

Subscribers: sbagaria, smahadevan, treaster

Maniphest Tasks: T36722

Differential Revision: https://phabricator.grailbio.com/D41472

fbshipit-source-id: a6cdc49

Summary: [We've started encountering this error](https://grailbio.slack.com/archives/CRRJ02JCQ/p1590079383265300) during S3 throttling.

Test Plan: Unit tests.

Reviewers: jcharumilind, smahadevan

Reviewed By: jcharumilind

Subscribers: soheil, jschellenberger

Differential Revision: https://phabricator.grailbio.com/D45625

fbshipit-source-id: 4a15ddf

Summary:
As mentioned in D44965, several people have asked in #eng-help about issues
with grail-access where deleting credentials before starting a new
authentication flow made things worse.

In the process, I've made some other improvements:
* Load principal with `agentlib` before calling `v23.Init`. This avoids some
  potentially confusing conflicts where `v23.Init` starts an agent and locks
  the principal so we can't load it.
* Respect `$V23_CREDENTIALS` is present. I think we should just delete our
  `-dir` flag, but I want to preserve compatibility for now.
* General cleanup: code deduplication, consistent logging, consistent contexts.

Test Plan:
Existing unit tests (from D44965), run on macOS and Linux with both Bazel and go test.

Also, I wrote a test script for manually testing the Google flow and executed on macOS and Linux.
Output on Linux (with redactions):

    $ cmd/grail-access/manual_google_test.bash
    Instructions:

    Running manual tests. If the script exits with a non-zero error code, the tests failed.
    You'll also be prompted to review output manually. If it doesn't look right, the tests failed.

    Using temporary directory for test: /tmp/tmp.PZSaBvnIrf
    Building grail-access for the test...

    Step 1/3: Starting grail-access Google authentication flow. Please complete it.

    ************************************************************************
    2020/05/28 06:08:37 INFO: Couldn't load principal from ./v23. Creating new one...
    I0528 06:08:37.176503   16864 google.go:67] listening: 127.0.0.1:35689
    The attempt to automatically open a browser failed. Please open the following link:

            https://accounts.google.com/o/oauth2/v2/auth?...

    Paste the received code and then press enter: ...

    Successfully applied new blessing:
    Public key: 6c:c8:a6:22:01:83:b5:61:f4:0e:f9:f0:3c:ab:73:d8
    ---------------- BlessingStore ----------------
    Default Blessings                v23.grail.com:google:joshnewman@grailbio.com
    Peer pattern                     Blessings
    ...                              v23.grail.com:google:joshnewman@grailbio.com
    ---------------- BlessingRoots ----------------
    Public key                                        Pattern
    98:61:28:77:eb:5f:97:7f:2c:ef:4d:1b:f1:4d:fd:c0   [v23.grail.com]
    Expires on 2020-06-04 06:08:45.544893802 +0000 UTC (in 168h0m0.002995278s)
    ************************************************************************

    Done with authentication flow.
    If it succeeded, you should lines like these above:
        Default Blessings      v23.grail.com:google:YOUR_USERNAME@grailbio.com
    and
        ...                    v23.grail.com:google:YOUR_USERNAME@grailbio.com
    and an expiration date in the future.

    Continue with next test? [Y]

    Step 2/3: Next, running the same flow, but automatically canceling.

    ************************************************************************
    I0528 06:08:49.313507   16964 google.go:67] listening: 127.0.0.1:38975
    The attempt to automatically open a browser failed. Please open the following link:

            https://accounts.google.com/o/oauth2/v2/auth?...

    Paste the received code and then press enter: ERROR: failed to fetch blessings: EOF
    ************************************************************************

    Step 3/3: Finally, make sure our Step 1 credentials survived.

    ************************************************************************
    Public key: 6c:c8:a6:22:01:83:b5:61:f4:0e:f9:f0:3c:ab:73:d8
    ---------------- BlessingStore ----------------
    Default Blessings                v23.grail.com:google:joshnewman@grailbio.com
    Peer pattern                     Blessings
    ...                              v23.grail.com:google:joshnewman@grailbio.com
    ---------------- BlessingRoots ----------------
    Public key                                        Pattern
    98:61:28:77:eb:5f:97:7f:2c:ef:4d:1b:f1:4d:fd:c0   [v23.grail.com]
    Expires on 2020-06-04 06:08:45.544893802 +0000 UTC (in 167h59m56.20723528s)
    ************************************************************************

    You should see the same blessing lines as in Step 1, and a consistent expiry time.
    If not, the tests failed.

    Cleaning up test directory: /tmp/tmp.PZSaBvnIrf

Reviewers: aeiser

Reviewed By: aeiser

Subscribers: jcharumilind, krestivo, smahadevan

Differential Revision: https://phabricator.grailbio.com/D45022

fbshipit-source-id: 76802df

…g and prepostdev)

Test Plan:
Check that the image is produced via webconsole

(It was been produced)

Reviewers: O34 Pre/Post, treaster, fdegiuli

Reviewed By: O34 Pre/Post, treaster

Subscribers: #lab_testing, smahadevan

Differential Revision: https://phabricator.grailbio.com/D45662

fbshipit-source-id: a47a020

Summary:
Remove terms and references that are based in racial segregation.

REF: T37372

Test Plan: unit tests related to allow/deny

Reviewers: bbentson, skrishnamurthy, ender

Reviewed By: skrishnamurthy

Subscribers: smahadevan

Maniphest Tasks: T37372

Differential Revision: https://phabricator.grailbio.com/D46020

fbshipit-source-id: ca05c4c

Summary: We want one image with both grail-access and grail-ticket so dumping grail-access binary into grail-ticket image.

Test Plan:
apply it to the trs server.
```
*******************************************************
*    WARNING: $V23_CREDENTIALS is not defined!        *
*******************************************************

How to fix this in bash: export V23_CREDENTIALS=/root/.v23

2020/06/11 17:39:01 INFO: Couldn't load principal from /root/.v23. Creating new one...
ERROR: failed to fetch blessings: grail-access:<rpc.Client>"/ticket-server.eng.grail.com:8102/blesser/ec2".BlessEc2: Error: launch time is too old: 2020-06-05 17:32:11 +0000 UTC should be within 1h0m0s
```
Returns the proper failure mode

Reviewers: fdegiuli, treaster

Reviewed By: fdegiuli

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D46068

fbshipit-source-id: 117bea8

Summary:
It accumulated many types and functions over the years and they were sometimes
interleaved in ways that didn't promote readability. I've split out a few files
mostly centered around particular operations.

This should be a pure refactoring. Almost all changes are just moving things
between files, with a few documentation changes and converting an unexported
function to a method (with a receiver). I expect no behavior changes.

Test Plan: Existing builds and tests. The test code was smaller and I didn't refactor it.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D46285

fbshipit-source-id: 53ac2a7

Summary:
I considered doing more sophisticated things (Cloudwatch, Prometheus push,
etc.), but I don't think it's clear yet how we want to use these (and how long
they'll stick around), so simple counting and logging seems like the best
short-term approach. It'll be easy to revert and replace later, or extend to
cover more S3 request types.

Test Plan:
Ran an S3-heavy program and observed logging like
    2020/06/22 23:40:07 s3file metrics: op:stat n:996 r:996/0/0/0 t:0/0/864/132/0/0/0 mib:0 [/min]
    2020/06/22 23:40:07 s3file metrics: op:read n:181692 r:3462/0/0/0 t:172248/5850/2562/1032/0/0/0 mib:10154 [/min]
    2020/06/22 23:40:17 s3file metrics: op:read n:13458 r:0/0/0/0 t:3234/10182/42/0/0/0/0 mib:13458 [/min]
    2020/06/22 23:40:27 s3file metrics: op:read n:13422 r:0/0/0/0 t:2442/10932/42/0/0/0/0 mib:13416 [/min]

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: smahadevan, jschellenberger

Differential Revision: https://phabricator.grailbio.com/D46329

fbshipit-source-id: f0ed231

Summary:
Experiments suggest this greatly reduces S3 request rate for sequential access
patterns.

Test Plan:
Read fragments files (which go through recordio magic checking, etc.) using a
few programs.

New simple unit tests as well as more extensive randomized operation generation
and comparison to `bytes.Buffer` (as a golden reference).

File reading benchmarks (that are quick and run on SSD, not S3, and thus are
unaffected by the original issue) are unchanged, which is good.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: smahadevan, jschellenberger

Differential Revision: https://phabricator.grailbio.com/D46330

fbshipit-source-id: 2acf8f3

…ounts to be validated as ticket-server users

Summary:
Allow the ticket-server to support multiple google domains when performing user verification and blessing.
Allow the ticket-server to support multiple google domains when performing google group lookups
Correct incorrect regex that did not escape special characters `.\([`.

FIXES T37864

Test Plan:
Run local ticket-server
Perform several calls to create a user, and use tickets

```
GB-C02YR4M9LVCG:sandbox.grail aeiser$ grail-access -blesser-google=/@6@wsh@:9002@@35741be6d925abac81c7878ff24e7dfa@s@v23.grail.com@@/blesser/google
I0618 22:23:21.863538   40060 google.go:67] listening: 127.0.0.1:53322
Opening "https://accounts.google.com/o/oauth2/v2/auth?access_type=online&client_id=27162366543-edih9cqc3t8p5hn9ord1k1n7h4oajfhm.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A53322&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&state=3f9d3f4f2bec9c70f6c7bd371593e0b1"...
Successfully applied new blessing:
Public key: cc:e5:97:7d:a8:f9:19:b1:fe:60:7d:5d:b3:98:29:26
​---------------- BlessingStore ----------------
Default Blessings                v23.grail.com:google:aeiser@contractors.grail.com
Peer pattern                     Blessings
...                              v23.grail.com:google:aeiser@contractors.grail.com
​---------------- BlessingRoots ----------------
Public key                                        Pattern
98:61:28:77:eb:5f:97:7f:2c:ef:4d:1b:f1:4d:fd:c0   [v23.grail.com]
Expires on 2020-06-25 22:24:42.691220945 -0700 PDT (in 167h59m59.981234945s)
```

```
GB-C02YR4M9LVCG:sandbox.grail aeiser$ grail-access -blesser-google=/@6@wsh@:9002@@35741be6d925abac81c7878ff24e7dfa@s@v23.grail.com@@/blesser/google
I0618 22:24:50.558327   40198 google.go:67] listening: 127.0.0.1:53714
Opening "https://accounts.google.com/o/oauth2/v2/auth?access_type=online&client_id=27162366543-edih9cqc3t8p5hn9ord1k1n7h4oajfhm.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A53714&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&state=c09ce502d5980d70a84d227579c9b209"...
ERROR: failed to fetch blessings: grail-access:<rpc.Client>"/@6@wsh@:9002@@35741be6d925abac81c7878ff24e7dfa@s@v23.grail.com@@/blesser/google".BlessGoogle: Error: ID token has a wrong hosted domain: got "sandbox.grail.com", want "grailbio.com,contractors.grail.com"
```

```
GB-C02YR4M9LVCG:sandbox.grail aeiser$ grail-access -blesser-google=/@6@wsh@:9002@@35741be6d925abac81c7878ff24e7dfa@s@v23.grail.com@@/blesser/google
I0618 22:24:58.766576   40203 google.go:67] listening: 127.0.0.1:53723
Opening "https://accounts.google.com/o/oauth2/v2/auth?access_type=online&client_id=27162366543-edih9cqc3t8p5hn9ord1k1n7h4oajfhm.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A53723&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&state=a34a4e69355f5f0a77e20c5d2df46722"...
Successfully applied new blessing:
Public key: cc:e5:97:7d:a8:f9:19:b1:fe:60:7d:5d:b3:98:29:26
​---------------- BlessingStore ----------------
Default Blessings                v23.grail.com:google:aeiser@grailbio.com
Peer pattern                     Blessings
...                              v23.grail.com:google:aeiser@grailbio.com
​---------------- BlessingRoots ----------------
Public key                                        Pattern
98:61:28:77:eb:5f:97:7f:2c:ef:4d:1b:f1:4d:fd:c0   [v23.grail.com]
Expires on 2020-06-25 22:25:02.026613997 -0700 PDT (in 167h59m59.992090997s)
```

Before aeiser@contractors.grail.com added to eng/dev
```
GB-C02YR4M9LVCG:sandbox.grail aeiser$ grail-access -blesser-google=/@6@wsh@:9002@@f085cdc756039daeccdb46a54cb2ac2a@s@v23.grail.com@@/blesser/google
Successfully applied new blessing:
Public key: cc:e5:97:7d:a8:f9:19:b1:fe:60:7d:5d:b3:98:29:26
​---------------- BlessingStore ----------------
Default Blessings                v23.grail.com:google:aeiser@contractors.grail.com
Peer pattern                     Blessings
...                              v23.grail.com:google:aeiser@contractors.grail.com
​---------------- BlessingRoots ----------------
Public key                                        Pattern
98:61:28:77:eb:5f:97:7f:2c:ef:4d:1b:f1:4d:fd:c0   [v23.grail.com]
Expires on 2020-06-25 22:41:20.033580147 -0700 PDT (in 167h59m59.993922147s)

GB-C02YR4M9LVCG:sandbox.grail aeiser$ grail-aws //@6@wsh@:9002@@f085cdc756039daeccdb46a54cb2ac2a@s@v23.grail.com@@/eng/dev/admin
no subcommand specified, inferring 'credentials' based on other args
E0618 22:41:33.777187   40932 conncache.go:319] Failed to resolve (wsh, :9002): lookup : no such host
ticket-server:"eng/dev/admin".Get: Access denied: ticket-server:"eng/dev/admin".Get: not authorized to call eng/dev/admin.Get: ticket-server:"eng/dev/admin".Get: [v23.grail.com:google:aeiser@contractors.grail.com] does not have Read access (rejected blessings: [])
```

After adding to eng/dev
```
GB-C02YR4M9LVCG:sandbox.grail aeiser$ grail-aws  /@6@wsh@:9002@@e10baa5fa1ae90e431e077e2ab7531af@s@v23.grail.com@@/eng/dev/admin
no subcommand specified, inferring 'credentials' based on other args
E0618 22:34:08.812232   40729 conncache.go:319] Failed to resolve (wsh, :9002): lookup : no such host
export AWS_ACCESS_KEY_ID=ASIAZAUXVPWNLVEPKWOU
export AWS_SECRET_ACCESS_KEY=XXXX
export AWS_SESSION_TOKEN=YYYY
```

Reviewers: #shared_infrastructure, btoledo

Reviewed By: #shared_infrastructure, btoledo

Subscribers: joshnewman, contractor-aeiser, btoledo, smahadevan

Maniphest Tasks: T37864

Differential Revision: https://phabricator.grailbio.com/D46325

fbshipit-source-id: c6a773e

Summary:
The [S3 performance guide](http://web.archive.org/web/20200624062712/https://docs.aws.amazon.com/AmazonS3/latest/dev/optimizing-performance-design-patterns.html) recommends using diverse S3 IPs (to distribute load and
reduce throttling). This revision adds a metric to count unique remote IPs used
by s3file's underlying HTTP clients. In one example program, I see no diversity:
    2020/06/24 06:35:38 s3file metrics: op:stat n:120 r:120/0/0/0 t:0/0/120/0/0/0/0 mib:0 [/min]
    2020/06/24 06:35:38 s3file metrics: op:read n:10020 r:552/0/0/0 t:9420/48/540/12/0/0/0 mib:309 [/min]
    2020/06/24 06:35:38 s3file metrics: op:dial 94/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0
Sometimes it's a little better:
    2020/06/24 06:35:10 s3file metrics: op:stat n:120 r:120/0/0/0 t:0/0/84/36/0/0/0 mib:0 [/min]
    2020/06/24 06:35:10 s3file metrics: op:read n:9168 r:552/0/0/0 t:8598/12/372/186/0/0/0 mib:282 [/min]
    2020/06/24 06:35:10 s3file metrics: op:dial 51/23/1/0/0/0/0/0/0/0/0/0/0/0/0/0
I plan to check on a few more important, often-throttled ones and then look into
fixing this.

A simple experiment program (resolvetest) suggests that we often get the same
result for a few seconds at a time, perhaps due to caching (in Go or Ubuntu or
AWS or elsewhere?), which may be useful if we try to fix this later:
    0.000000:       52.218.193.106
    0.027276:       52.218.193.106
    0.002374:       52.218.193.106
    0.004290:       52.218.193.106
    0.008292:       52.218.193.106
    0.016247:       52.218.193.106
    0.032277:       52.218.193.106
    0.064325:       52.218.193.106
    0.128351:       52.218.193.106
    0.256322:       52.218.193.106
    0.512338:       52.218.193.106
    1.024396:       52.218.193.106
    1.025318:       52.218.193.106
    1.024379:       52.218.192.194
    1.031849:       52.218.192.194
    1.024471:       52.218.246.131
    1.044295:       52.218.246.131
    1.024388:       52.218.246.131
    1.024385:       52.218.246.131
    1.024385:       52.218.216.186
    1.025139:       52.218.180.227
    1.025214:       52.218.180.227
    1.024409:       52.218.180.227
    1.024383:       52.218.180.227
    1.024413:       52.218.180.227
    1.024368:       52.218.246.211
    1.025224:       52.218.248.50
    1.025222:       52.218.222.27
    1.025249:       52.218.222.27
    1.024392:       52.218.176.163
    1.025139:       52.218.193.50

Test Plan:
Ran programs with the `-s3file.metric_http_addr` flag.
Ran resolvetest.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: smahadevan

Maniphest Tasks: T38104

Differential Revision: https://phabricator.grailbio.com/D46444

fbshipit-source-id: 067b1c6

Summary:
This updates bio/pileup and everything depending on it to use
bio/encoding/fasta, now that D46599 has made it painless.

Test Plan:
Existing tests pass.  end_to_end_model_training_contasource.sh
ran to completion on my adhoc.

Reviewers: sbagaria, ayip

Reviewed By: sbagaria

Subscribers: escott, cyakym, sbanville, cchang, smahadevan

Maniphest Tasks: T32685

Differential Revision: https://phabricator.grailbio.com/D46681

fbshipit-source-id: ea6bf9e

…session

Summary: Move ticket-based AWS configuration to common function in `cloud/awssession`, as discussed in D46367.

Test Plan: Run bigslice-demo example program.

Reviewers: joshnewman

Reviewed By: joshnewman

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D46792

fbshipit-source-id: a9745c7

Summary:
Adding a DeepSize() method which can compute the size of a complex struct.
Included also is an IntervalSet struct which could use some work.

Test Plan: extensive unit testing

Reviewers: joshnewman, jcharumilind

Reviewed By: joshnewman

Subscribers: smahadevan

Maniphest Tasks: T38940

Differential Revision: https://phabricator.grailbio.com/D47011

fbshipit-source-id: ebf03ed

Test Plan: Existing tests, and some queries with grail-file and grail-fuse.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: smahadevan

Maniphest Tasks: T30452

Differential Revision: https://phabricator.grailbio.com/D46790

fbshipit-source-id: 3441fff

Test Plan:
I used to see nothing from `ls /mnt/s3/.../dir_with_dir_marker`.
Now I see the correct contents.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: smahadevan

Maniphest Tasks: T30452

Differential Revision: https://phabricator.grailbio.com/D47379

fbshipit-source-id: afbb9ae

Summary:
Converting large tickets to JSON is an expensive operation and in some cases, causes grail-ticket to hang.

Only convert raw ticket data to JSON when the output is going to be in JSON format.

Test Plan:
Ran `grail-ticket tickets/aws-grail-edc-dev/global/secrets/summitRandomizationSchedule` (without the changes),
grail-ticket hangs.

Ran `bazel run //go/src/github.com/grailbio/base/cmd/grail-ticket -- tickets/aws-grail-edc-dev/global/secrets/summitRandomizationSchedule`
(with the changes), the ticket is pulled down successfully.

Reviewers: noah, aeiser

Reviewed By: aeiser

Subscribers: dgustafson, smahadevan, bbentson

Maniphest Tasks: T39843

Differential Revision: https://phabricator.grailbio.com/D47898

fbshipit-source-id: dee3c42

Summary:
The first version of demuxbcl was limited to 255 samples, since it was
convenient to use the "bytes" package and gunsafe.ExtendBytes() for a
few operations.  However, since we now know we need to demux larger
numbers of samples at a time, this revision changes the sampleId type
to uint16, which raises the limit to 65535 samples.

bytes.IndexByte() was actually moderately important for performance,
so I've added the analogous simd.IndexU16() function.

Test Plan:
Existing tests pass, and unit test and benchmark have been added for
simd.IndexU16().  Reran bio-demux-bcl on the dataset mentioned in
T34299, and verified results were unchanged.

Reviewers: ayip, ghorrell

Reviewed By: ayip

Subscribers: smahadevan, escott, sramani, nshojaee, cchang

Maniphest Tasks: T39842

Differential Revision: https://phabricator.grailbio.com/D48010

fbshipit-source-id: d47d8ab

As of grailbio@349e292, we are no longer Go 1.12-compatible. Go 1.12 is 3 versions out of date now, so we are fully dropping support.

This fixes part of the current build failures introduced by the last push from the GRAIL internal repository.

When we synced grailbio@4ef097c#diff-8b7b38c62b815a44970c5332bcadd718R64-R72 from our internal repository, we introduced a dependence on `ssh-keygen` support of the `-` argument to read from standard input. The version of `ssh-keygen` shipped on trusty does not support it. Upgrade our build to use xenial. (We may consider upgrading further, but I am trying to make the minimal changes to make our CI build succeed again).

…io#22)

Co-authored-by: Jaran Charumilind <jcharumilind@grailbio.com>

Summary:
Allow ticket-server to grant Kubernetes pods a v23 identity.

Given a request with a `ca.crt`, `namespace`, and service account `token`, query the Kubernetes cluster API server to verify the token. Verify that the request `ca.crt` is in the allow list of clusters.

Request times:
28 AWS accounts across 3 regions -> 23.89s
28 AWS accounts across 1 region (closest to what we will run in prod)-> 4.4s
5 AWS accounts across 3 regions -> 8.93s

Design Doc: https://docs.google.com/document/d/1-MxDlRT5JPucktIzjNfRFFmPjjXNjdmi5xjpjGQ2KU8/edit#

Closes T39592

Test Plan: Verify flow locally by granting a v23 identity from a local ticket-server using a Kubernetes cluster in the dev environment to authenticate the token.

Reviewers: O9 ticket-server, aeiser, O3 third-party, joshnewman, bbentson

Reviewed By: O9 ticket-server, O3 third-party, joshnewman, bbentson

Subscribers: joshnewman, bbentson, aeiser, jcharumilind, pgopal, dnicolaou, pboyapalli

Maniphest Tasks: T39579, T39592

Differential Revision: https://phabricator.grailbio.com/D47646

fbshipit-source-id: 3513e59

Summary:
This command will be run from a Kubernetes pod. It will request a v23 identity from the ticket-server using the /blesser/k8s endpoint and pass in caCrt, namespace, and service account token.

Design Doc: https://docs.google.com/document/d/1-MxDlRT5JPuckitIzjNfRFFmPjjXNjdmi5xjpjGQ2KU8/edit#

Closes T39590

Test Plan:
Verify flow locally.

Grant a v23 identity from a local ticket-server using a Kubernetes cluster in the dev environment to authenticate the token.

Subscribers

Reviewers: #security, aeiser, O9 ticket-server, joshnewman, bbentson

Reviewed By: #security, O9 ticket-server, joshnewman, bbentson

Subscribers: bbentson, joshnewman, smahadevan

Maniphest Tasks: T39590

Differential Revision: https://phabricator.grailbio.com/D47876

fbshipit-source-id: e77e5a8

Summary:
This adds SetInterval and ClearInterval functions which efficiently
set/clear a block of bits, along with NewSetBits()/NewClearBits()
convenience functions.

Test Plan: Unit test added.

Reviewers: ayip

Reviewed By: ayip

Subscribers: cchang, smahadevan

Maniphest Tasks: T20121

Differential Revision: https://phabricator.grailbio.com/D48407

fbshipit-source-id: 7aad6f4

Summary: agentlib is deprecated in the newest versions of vanadium. This diff will make upgrading our vanadium version easier.

Test Plan: Ran `grail-access` locally and updated unit tests.

Reviewers: #security, aeiser

Reviewed By: #security, aeiser

Subscribers: joshnewman, smahadevan

Differential Revision: https://phabricator.grailbio.com/D50485

fbshipit-source-id: 8e017ed

Summary:
Update the method comment on `Each` to warn developers about a potential
deadlock state when panics occur in concurrent goroutines.

Ref T43383

Test Plan: Verify that the code still compiles and bazel build passes.

Reviewers: arinehouse, ecarrel

Reviewed By: ecarrel

Subscribers: smahadevan

Tags: #software-only_change_control, #platform-wide

Maniphest Tasks: T43383

Differential Revision: https://phabricator.grailbio.com/D50868

fbshipit-source-id: f8d8ada

Summary:
This reverts commit 8e017ede14a0ea4fe00b428eee8fbae20a27306c.

This change is causing issues because we removed the v23agent without upgrading Vanadium. This change will be included in the diff where we upgrade the Vanadium version.

Test Plan: Unit tests.

Reviewers: aeiser

Reviewed By: aeiser

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D50927

fbshipit-source-id: dea4e06

Summary:
A new alternative to D46489 and D48243. This takes a different approach of
more explicitly managing transports tuned for S3 access.

Test Plan:
New unit tests.

Enabled new transport in (manually-run) integration tests, and ran them.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: aeiser, jschellenberger, smahadevan

Maniphest Tasks: T38104

Differential Revision: https://phabricator.grailbio.com/D48772

fbshipit-source-id: 6329221

Summary:
There was a bug where memory usage could be under counted.

Adds compaction to the intervalset.  At certain strategic times, adjacent
intervals in the interval set will be merged and de-duplicated.

Test Plan: updated unit tests

Reviewers: joshnewman

Reviewed By: joshnewman

Subscribers: anandi, jcharumilind, smahadevan

Maniphest Tasks: T38940

Differential Revision: https://phabricator.grailbio.com/D51318

fbshipit-source-id: abdd8ac

Summary:
Open Sourcing the common logging library so it can be used by services in github.com  directory, such as ticket-server.

Closes T43907.

Test Plan: Make sure builds pass.

Reviewers: O50 galleri/communications, O56 Ensemble backend, O15 scrms, #security, treaster, jgalilee, tule

Reviewed By: O50 galleri/communications, O56 Ensemble backend, O15 scrms, #security, jgalilee, tule

Subscribers: aeiser, jcharumilind, dnicolaou, pboyapalli, smahadevan

Maniphest Tasks: T43907

Differential Revision: https://phabricator.grailbio.com/D51322

fbshipit-source-id: d62074e

Summary: Switch ticket-server logger from vlog to our common, structured logger. Once we upgrade v.io, we should be able to include `requestID` in each log message that is unique to each server request. This will greatly improve ticket-server logging as it will allow us to corroborate multiple and simultaneous log messages.

Test Plan:
Instantiate local ticket-server, make requests against it, and verify logs and functionality.

Improved logging messages:
https://phabricator.grailbio.com/P1133

Reviewers: O9 ticket-server, aeiser

Reviewed By: O9 ticket-server, aeiser

Subscribers: jcharumilind, dnicolaou, pboyapalli, smahadevan

Maniphest Tasks: T43900

Differential Revision: https://phabricator.grailbio.com/D51308

fbshipit-source-id: 289ed44

Summary:
Add debug level logging, improve godoc documentation. These changes were requested by Fed.

Also add unit test for nil ctx.

Closes T44568

Test Plan: Update unit tests.

Reviewers: fdegiuli, treaster, adharwadkar

Reviewed By: fdegiuli

Subscribers: #security, smahadevan

Maniphest Tasks: T44568

Differential Revision: https://phabricator.grailbio.com/D51994

fbshipit-source-id: 45024c8

Summary:
Configure `tfwrap` with a profile specified in the classifier evaluation binary. We previously used configuration from `$HOME/grail/profile`, however this had at least two problems:

# It wasn't captured anywhere for reproducibility.
# It was a second place to deal with for configuration, in addition to `trainer_params`.

Since D51112, Chiron itself uses a `base/config` profile for configuration. In this revision, add `tfexec` things to that profile default, and now pass it as a command line argument via `tfwrap` for use by `tfexec`. This means that all profile configuration happens in `trainer_params`, and `$HOME/grail/profile` is defunct.

Profiles are only included if you specify `chiron_eval_binary(..., pass_profile = True, ...)`. In practice, you probably want to set this if and only if you are specifying a binary built with `tfexec_binary` as your trainer.

`tfwrap` no longer reads from `$HOME/grail/profile` to prevent accidental inclusion of configuration that prevents reproduction, however you can still invoke it directly with a `--profile your-profile` argument.

Test Plan: Run unit tests. Run with different configurations of the `sexandage` trainer.

Reviewers: O52 chiron infrastructure, joshnewman, soheil

Reviewed By: O52 chiron infrastructure, joshnewman

Subscribers: smahadevan

Tags: #chiron

Maniphest Tasks: T42162, T40897

Differential Revision: https://phabricator.grailbio.com/D51908

fbshipit-source-id: 72662d5

Summary:
Update the long description of grail-ticket to use `grail-ticket` instead of `grail ticket` and use the `tickets/eng/dev/aws`, which every engineer should have access to. Also re-generate doc.go to update.

Closes T44662

Test Plan: Careful eyes.

Reviewers: #security, aeiser

Reviewed By: #security, aeiser

Subscribers: smahadevan

Maniphest Tasks: T44662

Differential Revision: https://phabricator.grailbio.com/D52004

fbshipit-source-id: ac90ed7

Summary:
Create logging functions without a `ctx` to support logging in situations without a context.

Closes T44734

Test Plan: Update unit tests.

Reviewers: fdegiuli, treaster, adharwadkar

Reviewed By: fdegiuli

Subscribers: adharwadkar, smahadevan

Maniphest Tasks: T44734

Differential Revision: https://phabricator.grailbio.com/D52138

fbshipit-source-id: 4daca99

Summary:
Changed vlog to common logger
Where ever the context was not available in "main" like function, context.Background() context was used. In the NoCtx variations of the logger, the context.Background() is used (suggested by @treaster)

Test Plan: This should be printing the same message as original message. No change in functionality is expected.

Reviewers: noah, O56 Ensemble backend, fdegiuli

Reviewed By: O56 Ensemble backend, fdegiuli

Subscribers: noah, treaster, smahadevan, fdegiuli

Differential Revision: https://phabricator.grailbio.com/D51822

fbshipit-source-id: 776f0ce

Summary:
Upgrade from Vanadium v0.1.6 to v0.1.14.

Closes T42157.

Test Plan:
Verify Vanadium upgraade on local instances of ticket-server (Noah), LIMS (Richard), and Pipeline (Phillip).
https://docs.google.com/document/d/12ySNMFTUCY4F3rCmp2pJn2LzsgxZgn-JMbmijHycjCU/edit#

Testing done by @rhuang T43601

Reviewers: O9 ticket-server, O17 Instruments Service, O28 Grailbook Infra, O5 pipeline, O34 Pre/Post, O10 LIMS Server, O6 EDC, O3 third-party, O56 Ensemble backend, O57 Patient Service Backend, O50 galleri/communications, aeiser, tule, rhuang, O58 Shipment Service owner, pbohannon, byang, ajonnavithula

Reviewed By: O9 ticket-server, O17 Instruments Service, O28 Grailbook Infra, O5 pipeline, O34 Pre/Post, O10 LIMS Server, O6 EDC, O3 third-party, O56 Ensemble backend, O57 Patient Service Backend, O50 galleri/communications, aeiser, tule, rhuang, O58 Shipment Service owner, pbohannon, byang, ajonnavithula

Subscribers: fdegiuli, aahn, rhuang, krestivo, aeiser, pbohannon, jcharumilind, pgopal, smahadevan, dnicolaou, pboyapalli, #lab_testing

Maniphest Tasks: T42157

Differential Revision: https://phabricator.grailbio.com/D50466

fbshipit-source-id: 404e476

Summary:
I do not plan on landing this diff. It is simply here so we have a revert diff ready in case the Vanadium update diff D50466 causes unexpected issues.

Revert "v.io: update to v0.1.14"

This reverts commit e48affa9ec12d8152370e745be8c381acfd7651a.

Reviewers: O9 ticket-server, O17 Instruments Service, O50 galleri/communications!, O28 Grailbook Infra, O58 Shipment Service owner, O5 pipeline, O56 Ensemble backend, O34 Pre/Post, O57 Patient Service Backend, O10 LIMS Server, O6 EDC, O3 third-party, aeiser, rhuang, pbohannon, ajonnavithula, ecarrel

Reviewed By: O9 ticket-server, O17 Instruments Service, O28 Grailbook Infra, O58 Shipment Service owner, O5 pipeline, O56 Ensemble backend, O34 Pre/Post, O57 Patient Service Backend, O10 LIMS Server, O6 EDC, O3 third-party, aeiser, rhuang, pbohannon, ajonnavithula, ecarrel

Subscribers: aeiser, jcharumilind, smahadevan, dnicolaou, pboyapalli, #lab_testing

Differential Revision: https://phabricator.grailbio.com/D52269

fbshipit-source-id: 3d0d90c

Summary:
Upgrade from Vanadium to v0.1.15.

This diff is based on D50466. D50466 was reverted by D52269.

D50466 upgraded Vanadium to v0.1.14. There are no significant changes between v0.1.14 and v0.1.15, except the bugfix for an issue we discovered in the Vanadium source code: the `dir.lock` file in the credentials directory is not generated during the runtime factory initialization, even when the read-only flag is not set.

This diff will include the commit that fixes that bug so that the `dir.lock` file is properly created.

```
$ git diff D52508 D50466 --name-only

go/src/github.com/grailbio/base/go.mod
go/src/github.com/grailbio/base/go.sum
go/src/github.com/grailbio/bigmachine/go.sum
go/src/github.com/grailbio/bigslice/go.sum
go/src/github.com/grailbio/bio/go.sum
go/src/github.com/grailbio/diviner/go.sum
go/src/github.com/grailbio/doppelmark/go.sum
go/src/github.com/grailbio/gql/go.sum
go/src/github.com/grailbio/grit/go.sum
go/src/github.com/grailbio/hts/go.sum
go/src/github.com/grailbio/infra/go.sum
go/src/github.com/grailbio/ml/go.sum
go/src/github.com/grailbio/reflow/go.sum
go/src/github.com/grailbio/v23/factories/grail/go.mod
go/src/github.com/grailbio/v23/factories/grail/go.sum
go/src/grail.com/bigslice/sliceflags/v23.go
go/src/grail.com/billing/server/vanadium/vdl/account/account.vdl.go
go/src/grail.com/go.mod
go/src/grail.com/go.sum
go/src/grail.com/pipeline/director/services/release_test.go
go/src/grail.com/pipeline/director/services/worker_test.go
go/src/grail.com/pipeline/ifc/ifc.vdl.go
go/src/vendor/v.io/x/ref/lib/security/util.go
third-party/go/repositories.bzl
```

Full paste of diff D52508 D50466:
https://phabricator.grailbio.com/P1247

Test Plan:
There was already significant testing done for D50466 with Noah verifying the ticket-server behavior in staging environment, Richard on LIMS, and Philip on Pipeline.
Testing done by @rhuang T43601

The additional test for this diff is to verify that old credentials (geneated with old grail-access) is compatible with new grail-ticket. This was verified by Noah.

Closes T42157.

Reviewers: O9 ticket-server, O17 Instruments Service, O50 galleri/communications, O28 Grailbook Infra, O58 Shipment Service owner, O5 pipeline, O56 Ensemble backend, O34 Pre/Post, O57 Patient Service Backend, O10 LIMS Server, O6 EDC, O3 third-party, rhuang, aeiser, jgalilee, pbohannon, fdegiuli, lkong, ajonnavithula

Reviewed By: O9 ticket-server, O17 Instruments Service, O50 galleri/communications, O28 Grailbook Infra, O58 Shipment Service owner, O5 pipeline, O56 Ensemble backend, O34 Pre/Post, O57 Patient Service Backend, O10 LIMS Server, O6 EDC, O3 third-party, rhuang, aeiser, jgalilee, pbohannon, fdegiuli, lkong, ajonnavithula

Subscribers: joshnewman, jcharumilind, smahadevan, dnicolaou, pboyapalli, rhuang, #lab_testing

Maniphest Tasks: T42157

Differential Revision: https://phabricator.grailbio.com/D52508

fbshipit-source-id: a445ca1

Summary: With the v23 update to v0.1.15, the vcontext should now include a request id per v23 request.

Test Plan: Verified in local instantiation of ticket-server's logs.

Reviewers: fdegiuli, treaster, aeiser

Reviewed By: aeiser

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D52769

fbshipit-source-id: 2952ca9

Summary:
Update the `grail-ticket`  to support a flag, `-list`, that lists all the tickets accessible to the user.

To support this on the ticket-server side, a new function, `List()`, was added. `List()` iterates over all tickets and queries Google Groups API to see if the user has access. It then returns a sorted list of accessible tickets.

Example:
```
$ grail-ticket -list tickets
aws-grail-eng-sandbox/admin/aws
aws-grail-eng-sandbox/test/aws
eng/dev/admin
eng/dev/aws
eng/go-testing/aws
eng/go-testing/aws.perm
security/admin/aws
security/veeva/sbx-svc-logging-password
security/veeva/sbx-svc-logging-user
ti-apps/admin/aws
ti-apps/ticket-server/aws
```

From my local testing, the command takes 55-60s.

Test Plan: Instantiated ticket-server locally and ran `grail-ticket -list` against it.

Reviewers: O9 ticket-server, #security, aeiser

Reviewed By: O9 ticket-server, #security, aeiser

Subscribers: aeiser, smahadevan

Differential Revision: https://phabricator.grailbio.com/D52062

fbshipit-source-id: 6afaffc

Summary:
Automatically find identical strings and save memory by using the same underlying memory.
Does not work on unexported fields, which is a fairly big loophole.

For exampleSet, string interning saves 6% of memory as measured by DeepSize()

Test Plan:
unit tests for string interning method.
tested on exampleSet showing 6% memory savings

Reviewers: joshnewman

Reviewed By: joshnewman

Subscribers: afields, sbagaria, kdavydov, jcharumilind, smahadevan

Differential Revision: https://phabricator.grailbio.com/D52589

fbshipit-source-id: 97679c1

Summary: Previously, registering the empty string "" endpoint was causing glob to fail. So, we will change the endpoint to be "list" and append "list" to the path.

Test Plan:
Deployed locally and to staging tickets-testing ticket-server and verified flow with:
```
grail-ticket -list tickets-testing
```
Also verified that the glob functionality still works:
```
namespace glob tickets-testing/...
```

Reviewers: O9 ticket-server, bbentson

Reviewed By: O9 ticket-server, bbentson

Subscribers: aeiser, bbentson, smahadevan

Differential Revision: https://phabricator.grailbio.com/D53226

fbshipit-source-id: 85c3615

Summary:
D53169 introduces a check for this error condition.  I'm exporting the
error string to make it less likely the check will unexpectedly stop
working.

Test Plan: Existing tests pass.

Reviewers: osakarya

Reviewed By: osakarya

Subscribers: cchang, smahadevan

Differential Revision: https://phabricator.grailbio.com/D53533

fbshipit-source-id: cf99570

Summary:
Change the automatically logged `caller` field from full file path to only file name.

The impetus for this change is that the caller root will be different depending on the system. For example, the tests will pass with `bazel test`, but fail with `go test`. We could possibly configure the full file path to be an option in the future.

Github Issue: grailbio#28

Test Plan: Update unit tests.

Reviewers: fdegiuli, msabherwal

Reviewed By: fdegiuli

Subscribers: jcharumilind, smahadevan

Differential Revision: https://phabricator.grailbio.com/D53636

fbshipit-source-id: 0a74d6d

Summary:
This diff refactors the logging system to support an interface. This gives more flexibility to users, specifically the ability to mock the logger.

With this refactor, we now support two logging flows. The first is the current flow with global, structless functions. The second is a flow that requires instantiation of the logger via `NewLogger()`.

Test Plan: Update unit tests and include tests for both flows.

Reviewers: msabherwal, fdegiuli

Reviewed By: msabherwal, fdegiuli

Subscribers: aeiser, jcharumilind, smahadevan

Differential Revision: https://phabricator.grailbio.com/D53655

fbshipit-source-id: 0a27a40

Summary:
Reduce dump-related logging noise by:

* Making the shortened CPU-profile message a `debug` level log, as it's generally not a very big deal.
* Eliminating the message about missing bigmachine profiles when no machines exist yet.  It's not an error for there to be no machines, so we no longer log about it.

Test Plan: Run `bazel run //go/src/grail.com/cmd/bigslice-demo -- fragments -bigslice-system=ec2:instance=m5.xlarge -bigslice-load-factor=1.0 -bigslice-parallelism=96`, and see quieter logs.

Reviewers: joshnewman

Reviewed By: joshnewman

Subscribers: smahadevan

Maniphest Tasks: T45889

Differential Revision: https://phabricator.grailbio.com/D53850

fbshipit-source-id: db620b4

After syncing with the GRAIL internal repo, there were build failures due to stale `go.mod` entries.  I have updated them with internal changes to fix those build failures and run `go mod tidy` to eliminate any unused entries.

Unit tests succeed.

Summary: Renaming a variable from "bytes" to "numBytes" to avoid any potential confusion

Test Plan: existing unit tests are sufficient.

Reviewers: kdavydov

Reviewed By: kdavydov

Subscribers: smahadevan

Maniphest Tasks: T46101

Differential Revision: https://phabricator.grailbio.com/D53971

fbshipit-source-id: ed29f55

Summary:
Kill `eventlog` logs written by workers, e.g.
```
https://ec2-18-237-227-158.us-west-2.compute.amazonaws.com/i-0bd5f973615fde562/: 2021/01/27 22:03:37 eventlog: CloudWatch Logs group/stream: bigslice-event/543000147~unknown@grailbio.com~20210127T220337.087+0000
```
This happens because we logged before the call to `github.com/grailbio/bigmachine.Start`.  This now happens after that call, so it only executes on the driver.

Test Plan: Run the `/cmd/bigslice-demo -- fragments -max-samples=50 -bigslice-system=ec2:instance=m5.xlarge -bigslice-load-factor=1.0 -bigslice-parallelism=8` and see that there are no confusing `eventlog` log messages in the output.

Reviewers: joshnewman

Reviewed By: joshnewman

Subscribers: smahadevan

Maniphest Tasks: T45889

Differential Revision: https://phabricator.grailbio.com/D54048

fbshipit-source-id: 923fde6

Summary: To allow reusing this package elsewhere.

Test Plan:
Existing unit and integration tests:
    $ go test . -s3-bucket grail-joshnewman-test -s3-dir s3test
    ok      github.com/grailbio/base/file/s3file    23.392s

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D54414

fbshipit-source-id: 4c59c8c

Summary: This will open a new window in a new profile in Google Chrome when AWS_ENV is set

Test Plan:
`bazel build :grail-aws` then
`AWS_ENV=mrd $(bazel info bazel-bin)/go/src/grail.com/cmd/grail-aws/grail-aws tickets/grail-prod-mrd/admin/aws webconsole`

Reviewers: spolakh, aeiser

Reviewed By: aeiser

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D54579

fbshipit-source-id: 418081e

Summary: Fix header spec.

Test Plan: - NA

Reviewers: pbohannon

Reviewed By: pbohannon

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D54768

fbshipit-source-id: c76e5a5

Summary: Re-use a single timer for efficiency in various places.

Test Plan: Unit tests

Reviewers: pboyapalli

Reviewed By: pboyapalli

Subscribers: dnicolaou

Differential Revision: https://phabricator.grailbio.com/D55028

fbshipit-source-id: b55cd37

Summary: Currently, the exponential backoff policy will loop indefinitely. This diff adds an exponential backoff policy that will timeout after the max duration.

Test Plan: Add unit tests for `BackoffWithTimeout` policy.

Reviewers: jhughes

Reviewed By: jhughes

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D55184

fbshipit-source-id: f680d31

Summary:
This option can provide a formatting directive. One use case is to
print floating point numbers with a specified precision.

Test Plan: See included testable example.

Reviewers: jcharumilind, cchang

Reviewed By: jcharumilind, cchang

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D55258

fbshipit-source-id: bf058c4

Summary:
Currently, due to output being piped to both stderr and stdout, the logger package duplicates logs when run within a Kubernetes pod. The logger package does not duplicate logs when run within an EC2 instance because only stderr is captured.

This diff will fix the issue of logs being duplicated when run within a Kubernetes pod by adding the ability to configure the output path.

This diff also lays the foundation of customizing the logger. In a separate diff, I will add the ability to set the log level, which currently defaults to `Debug`.

Test Plan: Update unit tests.

Reviewers: fdegiuli

Reviewed By: fdegiuli

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D55332

fbshipit-source-id: c75cb96

Summary: Applications based on recordio can benefit from knowing an estimate of an upper bound on the space required to store the records for a given payload. We add an estimator to serve this.

Test Plan: Add new unit tests.

Reviewers: smahadevan, tcho

Reviewed By: smahadevan

Subscribers: smahadevan

Maniphest Tasks: T47160

Differential Revision: https://phabricator.grailbio.com/D55219

fbshipit-source-id: 5fd3e9a

Summary: We can refactor BackoffWithTimeout to reuse Backoff and MaxTries.

Test Plan: Unit tests.

Reviewers: smahadevan

Reviewed By: smahadevan

Subscribers: jhughes, ajonnavithula, smahadevan

Differential Revision: https://phabricator.grailbio.com/D55260

fbshipit-source-id: 7b673c8

Summary:
Add logging level to logger config. This will set the logging level of the logger. We will also change the default logging level from `Debug` to `Info`.

For example, setting the logging level to `Warn` will ignore all `Debug` and `Info` messages.

Test Plan: Add unit test.

Reviewers: fdegiuli

Reviewed By: fdegiuli

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D55383

fbshipit-source-id: 0bd4a5c

Summary:
This change adds a `SkipHeader` option that allows for users to create a writer that skips writing a header and begins in the `wStateWritingBody` state. This can be used to continue writing from where a previous writer may have left off.
This change also adds a `KeyTrailer` option that is needed when the recordio file contains a trailer. Previously the only way to set this was by explicitly adding this as a header.

Test Plan: Unit tests have been added for the normal write then read functionality as well as testing functioning with transformers.

Reviewers: kshashidhar, smahadevan

Reviewed By: kshashidhar, smahadevan

Subscribers: pbohannon

Maniphest Tasks: T47010

Differential Revision: https://phabricator.grailbio.com/D54947

fbshipit-source-id: be5a000

Summary: To allow more efficient single-entry reads, like from S3.

Test Plan: Updated existing unit tests. Generated autolog dump outputs with `bigslice-demo wait`, and they were valid files.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: jschellenberger, smahadevan

Differential Revision: https://phabricator.grailbio.com/D55521

fbshipit-source-id: 3a0a94a

Summary:
Use new rule to manage generation of fragments code. I think this will be helpful for maintainability (for non-github.com code).

If this works well, future changes can expand it to the rest of the generated fragments code, protobuf stubs (so we don't need to handle those separately in Arcanist), SQL schemas, VDL, cmdline doc.go, etc.

Test Plan: "Accidentally" modified a generated file, got a prompt to regenerate it, and then fixed it.

Reviewers: jcharumilind

Reviewed By: jcharumilind

Subscribers: noah, smahadevan, sbagaria

Differential Revision: https://phabricator.grailbio.com/D55482

fbshipit-source-id: 7e1908a

Summary:
Add Retry Logic like lockbox in Patient Service

Jira Task: CPP-1140

Test Plan:
All existing unit tests pass.
Added a new unit test.

Reviewers: ajonnavithula, atao, adevarmanai, jhughes, O57 Patient Service Backend

Reviewed By: adevarmanai, O57 Patient Service Backend

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D55841

fbshipit-source-id: cc9a59a

… for hash

Summary: Update Lockbox to use new pattern for retry

Test Plan: Same test cases passed, also update hash to retry slower

Reviewers: ajonnavithula, atao, adevarmanai, jhughes, ehu

Reviewed By: adevarmanai

Subscribers: smahadevan, O59 lockbox

Differential Revision: https://phabricator.grailbio.com/D56350

fbshipit-source-id: 7832ef1

Test Plan: CI totally has our backs, brah

Reviewers: noah, fdegiuli

Reviewed By: noah, fdegiuli

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56340

fbshipit-source-id: 42c6b9d

Summary: Enable setting the log level using an environment variable, `LOG_LEVEL`. The supported values will be "debug", "DEBUG", "info", "INFO", "warn", "WARN", "error", and "ERROR". Setting the environment variable LOG_LEVEL will override Config.Level.

Test Plan: Unit test.

Reviewers: O63 logger, jkimmel

Reviewed By: jkimmel

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56405

fbshipit-source-id: 170a37b

Summary:
* Change log lines that should be debug level to debug level.
* Standardize logging across ticket-server requests

Test Plan: Spun up local ticket-server and made requests against it.

Reviewers: O9 ticket-server, aeiser

Reviewed By: O9 ticket-server, aeiser

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56364

fbshipit-source-id: 698e00b

Summary: Add logging to ticket-server list endpoint.

Test Plan:
Spun up ticket-server locally and sent request to list endpoint.
```
{
  "level": "info",
  "msg": "list request",
  "caller": "list.go:19",
  "ts": "2021-03-18T23:15:21.061095800Z",
  "v23RequestID": "cf80131c-883f-11eb-bde0-0242ac110003",
  "endpoint": "list",
  "blessing": "v23.grail.com:google:ntseng@grailbio.com",
  "ticket": "list"
}
```

Note: the "ticket" k/v pair field name is a little funky because of how we implemented the list functionality.

Reviewers: O9 ticket-server, aeiser

Reviewed By: O9 ticket-server, aeiser

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56359

fbshipit-source-id: 1cd1290

Summary:
Currently, the default log level is `Info`. This is because the Zap library sets the default to `Info` and an empty level corresponds to `Info`. Any instantiation of `Config` will have `Level` as `Info` instead of `Debug`.

This is a messy problem and I think this is the best solution. If you can think of a better one, please let me know!

Test Plan: .

Reviewers: fdegiuli, jkimmel

Reviewed By: jkimmel

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56367

fbshipit-source-id: 3ba967e

…d rationale flag cli flag

Summary:
Add additional controls to ticket-server tickets, specifically Rationale, PagerDutyId, and TicketId. This will leverage a new endpoint, GetWithArgs. The commands used to interface with ticket-server, cloudkey, grail-aws, and grail-ticket, have been updated with a flag to set Rationale. More support for other Controls will come in the future.

Design Doc: https://docs.google.com/document/d/1dHroVUa52yJrJZGlb_jdEN8j3HGQttjQEPRFffxdEKk/edit#heading=h.a28np8vdza4

Test Plan: Spun up ticket-server locally and tested fetching tickets with and without a rationale using all three commands.

Reviewers: O9 ticket-server, aeiser

Reviewed By: O9 ticket-server, aeiser

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D54875

fbshipit-source-id: f7a92fd

Summary: Return an `error` with `log.Error*()` for convenience. This change should be backwards compatible and not cause any lint issues. Users can optionally use or ignore the returned `error`.

Test Plan: .

Reviewers: O63 logger, fdegiuli

Reviewed By: O63 logger, fdegiuli

Subscribers: fdegiuli, smahadevan

Differential Revision: https://phabricator.grailbio.com/D56516

fbshipit-source-id: 8015af8

Summary: Let WaitForFn take in multiple output length

Test Plan: New unit test testing multiple output length

Reviewers: jaliu

Reviewed By: jaliu

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56586

fbshipit-source-id: a44f802

Summary: to include a section about `Error` family of functions returning an `error`.

Test Plan: .

Reviewers: fdegiuli, O63 logger, jkimmel

Reviewed By: jkimmel

Subscribers: jkimmel, smahadevan

Differential Revision: https://phabricator.grailbio.com/D56753

fbshipit-source-id: 29c91f5

Summary: include more details about the returned error (missed comment from previous diff)

Test Plan: .

Reviewers: O63 logger, fdegiuli, jkimmel

Reviewed By: O63 logger, fdegiuli, jkimmel

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56769

fbshipit-source-id: db46756

Summary:
This reverts commit 8015af8220d1fb068934a6ad77e53879fc795508.

They will be replaced with a new family of functions, `Error*AndReturn`,
that return the log message (without keys and values) as a string.

Test Plan: Update unit tests.

Reviewers: O63 logger, O56 Ensemble backend, fdegiuli, jkimmel

Reviewed By: O63 logger, O56 Ensemble backend, fdegiuli

Subscribers: smahadevan

Differential Revision: https://phabricator.grailbio.com/D56965

fbshipit-source-id: a283619