Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Official Buf Plugin #573

Merged
merged 3 commits into from
May 27, 2022
Merged

feat: Official Buf Plugin #573

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a4e13d7
Added CI workflow for publishing Buf plugin
Jake-RoundrockIO May 26, 2022
e68b708
Updated some integration files due to build:test
Jake-RoundrockIO May 26, 2022
4eea566
Updated buf.yml to utilize variables for plugin owner
Jake-RoundrockIO May 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/buf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: Buf Plugin

on:
push:
tags:
- v*.*.*

jobs:
push_to_buf_registry:
name: Push Docker image to Buf Registry
runs-on: ubuntu-latest
steps:
- name: Check out the repo
uses: actions/checkout@v3

- name: Set output
id: vars
run: echo ::set-output name=tag::${GITHUB_REF#refs/*/}

- name: Log in to Buf Plugin Registry
uses: docker/login-action@v2
with:
registry: plugins.buf.build
username: ${{ secrets.BUF_USERNAME }}
password: ${{ secrets.BUF_PASSWORD }}

- name: Build Plugin image
env:
RELEASE_VERSION: ${{ steps.vars.outputs.tag }}
run: docker build
--file ts-proto.Dockerfile
--tag plugins.buf.build/${{ secrets.BUF_USERNAME }}/ts-proto:${RELEASE_VERSION}-0
"."

- name: Push to Buf Registry
env:
RELEASE_VERSION: ${{ steps.vars.outputs.tag }}
run: docker push plugins.buf.build/${{ secrets.BUF_USERNAME }}/ts-proto:${RELEASE_VERSION}-0
103 changes: 57 additions & 46 deletions README.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,17 @@
- [Table of contents](#table-of-contents)
- [Overview](#overview)
- [QuickStart](#quickstart)
- [Buf](#buf)
- [Buf](#buf)
- [Goals](#goals)
- [Example Types](#example-types)
- [Highlights](#highlights)
- [Auto-Batching / N+1 Prevention](#auto-batching--n1-prevention)
- [Usage](#usage)
- [Supported options](#supported-options)
- [Only Types](#only-types)
- [NestJS Support](#nestjs-support)
- [Watch Mode](#watch-mode)
- [Basic gRPC implementation](#basic-grpc-implementation)
- [Supported options](#supported-options)
- [Only Types](#only-types)
- [NestJS Support](#nestjs-support)
- [Watch Mode](#watch-mode)
- [Basic gRPC implementation](#basic-grpc-implementation)
- [Sponsors](#sponsors)
- [Development](#development)
- [Assumptions](#assumptions)
Expand Down Expand Up @@ -97,7 +97,16 @@ plugins:
out: ../gen/ts
strategy: all
path: ../node_modules/ts-proto/protoc-gen-ts_proto
```
```

You can also use the official plugin published to the Buf Registry.

```yaml
version: v1
plugins:
- remote: buf.build/roundrock-io/plugins/ts-proto
out: ../gen/ts
```

# Goals

Expand Down Expand Up @@ -174,7 +183,7 @@ creating a class and calling the right getters/setters.

(Configurable with the `useDate` parameter.)

- `fromJSON`/`toJSON` use the [proto3 canonical JSON encoding format](https://developers.google.com/protocol-buffers/docs/proto3#json) (e.g. timestamps are ISO strings), unlike [`protobufjs`](https://github.com/protobufjs/protobuf.js/issues/1304).
- `fromJSON`/`toJSON` use the [proto3 canonical JSON encoding format](https://developers.google.com/protocol-buffers/docs/proto3#json) (e.g. timestamps are ISO strings), unlike [`protobufjs`](https://github.com/protobufjs/protobuf.js/issues/1304).

- ObjectIds can be mapped as `mongodb.ObjectId`

Expand Down Expand Up @@ -220,7 +229,7 @@ protoc --plugin=node_modules/ts-proto/protoc-gen-ts_proto ./batching.proto -I.

`ts-proto` can also be invoked with [Gradle](https://gradle.org) using the [protobuf-gradle-plugin](https://github.com/google/protobuf-gradle-plugin):

``` groovy
```groovy
protobuf {
plugins {
// `ts` can be replaced by any unused plugin name, e.g. `tsproto`
Expand Down Expand Up @@ -366,19 +375,19 @@ Generated code will be placed in the Gradle build directory.

- With `--ts_proto_opt=metadataType=Foo@./some-file`, ts-proto add a generic (framework-agnostic) metadata field to the generic service definition.

- With `--ts_proto_opt=outputServices=generic-definitions,outputServices=default`, ts-proto will output both generic definitions and interfaces. This is useful if you want to rely on the interfaces, but also have some reflection capabilities at runtime.
- With `--ts_proto_opt=outputServices=generic-definitions,outputServices=default`, ts-proto will output both generic definitions and interfaces. This is useful if you want to rely on the interfaces, but also have some reflection capabilities at runtime.

- With `--ts_proto_opt=outputServices=false`, or `=none`, ts-proto will output NO service definitions.

- With `--ts_proto_opt=emitImportedFiles=false`, ts-proto will not emit `google/protobuf/*` files unless you explicit add files to `protoc` like this
`protoc --plugin=./node_modules/.bin/protoc-gen-ts_proto my_message.proto google/protobuf/duration.proto`
`protoc --plugin=./node_modules/.bin/protoc-gen-ts_proto my_message.proto google/protobuf/duration.proto`

- With `--ts_proto_opt=fileSuffix=<SUFFIX>`, ts-proto will emit generated files using the specified suffix. A `helloworld.proto` file with `fileSuffix=.pb` would be generated as `helloworld.pb.ts`. This is common behavior in other protoc plugins and provides a way to quickly glob all the generated files.
- With `--ts_proto_opt=fileSuffix=<SUFFIX>`, ts-proto will emit generated files using the specified suffix. A `helloworld.proto` file with `fileSuffix=.pb` would be generated as `helloworld.pb.ts`. This is common behavior in other protoc plugins and provides a way to quickly glob all the generated files.

- With `--ts_proto_opt=enumsAsLiterals=true`, the generated enum types will be enum-ish object with `as const`.

- With `--ts_proto_opt=useExactTypes=false`, the generated `fromPartial` method will not use Exact types.

The default behavior is `useExactTypes=true`, which makes `fromPartial` use Exact type for its argument to make TypeScript reject any unknown properties.

- With `--ts_proto_opt=unknownFields=true`, all unknown fields will be parsed and output as arrays of buffers.
Expand All @@ -396,6 +405,7 @@ Generated code will be placed in the Gradle build directory.
Note that, as indicated, this means Object.keys will not include set-by-default fields, so if you have code that iterates over messages keys in a generic fashion, it will have to also iterate over keys inherited from the prototype.

### Only Types

If you're looking for `ts-proto` to generate only types for your Protobuf types then passing all three of `outputEncodeMethods`, `outputJsonMethods`, and `outputClientImpl` as `false` is probably what you want, i.e.:

`--ts_proto_opt=onlyTypes=true`.
Expand All @@ -417,15 +427,15 @@ If you want to run `ts-proto` on every change of a proto file, you'll need to us

`ts-proto` is RPC framework agnostic - how you transmit your data to and from
your data source is up to you. The generated client implementations all expect
a `rpc` parameter, which type is defined like this:
a `rpc` parameter, which type is defined like this:

```ts
interface Rpc {
request(service: string, method: string, data: Uint8Array): Promise<Uint8Array>;
}
```

If you're working with gRPC, a simple implementation could look like this:
If you're working with gRPC, a simple implementation could look like this:

```ts
type RpcImpl = (service: string, method: string, data: Uint8Array) => Promise<Uint8Array>;
Expand Down Expand Up @@ -455,14 +465,14 @@ const sendRequest: RpcImpl = (service, method, data) => {
});
};

const rpc: Rpc = { request: sendRequest }
const rpc: Rpc = { request: sendRequest };
```

# Sponsors

Kudos to our sponsors:

* [ngrok](https://ngrok.com) funded ts-proto's initial grpc-web support.
- [ngrok](https://ngrok.com) funded ts-proto's initial grpc-web support.

If you need ts-proto customizations or priority support for your company, you can ping me at [via email](mailto:stephen.haberman@gmail.com).

Expand All @@ -481,9 +491,10 @@ The commands below assume you have **Docker** installed. To use a **local** copy
- Run `yarn install` to install the dependencies.
- Run `yarn build:test` or `yarn build:test:local` to generate the test files.
> _This runs the following commands:_
> - `proto2bin` — Converts integration test `.proto` files to `.bin`.
> - `bin2ts` — Runs `ts-proto` on the `.bin` files to generate `.ts` files.
> - `proto2pbjs` — Generates a reference implementation using `pbjs` for testing compatibility.
>
> - `proto2bin` — Converts integration test `.proto` files to `.bin`.
> - `bin2ts` — Runs `ts-proto` on the `.bin` files to generate `.ts` files.
> - `proto2pbjs` — Generates a reference implementation using `pbjs` for testing compatibility.
- Run `yarn test`

**Workflow**
Expand All @@ -494,16 +505,16 @@ The commands below assume you have **Docker** installed. To use a **local** copy
_Since the proto files were not changed, you only need to regenerate the typescript files._
- Run `yarn test` to verify the typescript files are compatible with the reference implementation, and pass other tests.
- Updating or adding `.proto` files in the integration directory:
- Run `yarn watch` to automatically regenerate test files when proto files change.
- Or run `yarn build:test` to regenerate all integration test files.
- Run `yarn watch` to automatically regenerate test files when proto files change.
- Or run `yarn build:test` to regenerate all integration test files.
- Run `yarn test` to retest.

**Contributing**

- Run `yarn build:test` and `yarn test` to make sure everything works.
- Run `yarn prettier` to format the typescript files.
- Commit the changes:
- Also include the generated `.bin` files for the tests where you added or modified `.proto` files.
- Also include the generated `.bin` files for the tests where you added or modified `.proto` files.
> These are checked into git so that the test suite can run without having to invoke the `protoc` build chain.
- Also include the generated `.ts` files.
- Create a pull request
Expand Down Expand Up @@ -594,7 +605,7 @@ Foo.encode({ bar: '' }); // => { }, writes an empty Foo object, in protobuf bina
Reading JSON will also initialize the default values. Since senders may either omit unset fields, or set them to the default value, use `fromJSON` to normalize the input.

```typescript
Foo.fromJSON({ }); // => { bar: '' }
Foo.fromJSON({}); // => { bar: '' }
Foo.fromJSON({ bar: '' }); // => { bar: '' }
Foo.fromJSON({ bar: 'baz' }); // => { bar: 'baz' }
```
Expand All @@ -603,21 +614,21 @@ When writing JSON, `ts-proto` currently does **not** normalize message when conv

```typescript
// Current ts-proto behavior
Foo.toJSON({ }); // => { }
Foo.toJSON({}); // => { }
Foo.toJSON({ bar: undefined }); // => { }
Foo.toJSON({ bar: '' }); // => { bar: '' } - note: this is the default value, but it's not omitted
Foo.toJSON({ bar: 'baz' }); // => { bar: 'baz' }
```

```typescript
// Possible future behavior, where ts-proto would normalize message
Foo.toJSON({ }); // => { }
Foo.toJSON({}); // => { }
Foo.toJSON({ bar: undefined }); // => { }
Foo.toJSON({ bar: '' }); // => { } - note: omitting the default value, as expected
Foo.toJSON({ bar: 'baz' }); // => { bar: 'baz' }
```

- Please open an issue if you need this behavior.
- Please open an issue if you need this behavior.

# Well-Known Types

Expand All @@ -626,25 +637,24 @@ Their interpretation is defined by the Protobuf specification, and libraries are

`ts-proto` currently automatically converts these messages to their corresponding native types.

* [google.protobuf.BoolValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#boolvalue) &lrarr; `boolean`
* [google.protobuf.BytesValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#bytesvalue) &lrarr; `Uint8Array`
* [google.protobuf.DoubleValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#doublevalue) &lrarr; `number`
* [google.protobuf.FieldMask](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#fieldmask) &lrarr; `string[]`
* [google.protobuf.FloatValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#floatvalue) &lrarr; `number`
* [google.protobuf.Int32Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#int32value) &lrarr; `number`
* [google.protobuf.Int64Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#int64value) &lrarr; `number`
* [google.protobuf.ListValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#listvalue) &lrarr; `any[]`
* [google.protobuf.UInt32Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#uint32value) &lrarr; `number`
* [google.protobuf.UInt64Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#uint64value) &lrarr; `number`
* [google.protobuf.StringValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#stringvalue) &lrarr; `string`
* [google.protobuf.Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#value) &lrarr; `any` (i.e. `number | string | boolean | null | array | object`)
* [google.protobuf.Struct](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#struct) &lrarr; `{ [key: string]: any }`
- [google.protobuf.BoolValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#boolvalue) &lrarr; `boolean`
- [google.protobuf.BytesValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#bytesvalue) &lrarr; `Uint8Array`
- [google.protobuf.DoubleValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#doublevalue) &lrarr; `number`
- [google.protobuf.FieldMask](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#fieldmask) &lrarr; `string[]`
- [google.protobuf.FloatValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#floatvalue) &lrarr; `number`
- [google.protobuf.Int32Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#int32value) &lrarr; `number`
- [google.protobuf.Int64Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#int64value) &lrarr; `number`
- [google.protobuf.ListValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#listvalue) &lrarr; `any[]`
- [google.protobuf.UInt32Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#uint32value) &lrarr; `number`
- [google.protobuf.UInt64Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#uint64value) &lrarr; `number`
- [google.protobuf.StringValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#stringvalue) &lrarr; `string`
- [google.protobuf.Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#value) &lrarr; `any` (i.e. `number | string | boolean | null | array | object`)
- [google.protobuf.Struct](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#struct) &lrarr; `{ [key: string]: any }`

## Wrapper Types

Wrapper Types are messages containing a single primitive field, and can be imported in `.proto` files with `import "google/protobuf/wrappers.proto"`.


Since these are _messages_, their default value is `undefined`, allowing you to distinguish unset primitives from their default values, when using Wrapper Types.
`ts-proto` generates these fields as `<primitive> | undefined`.

Expand All @@ -671,13 +681,13 @@ interface ExampleMessage {
When encoding a message the primitive value is converted back to its corresponding wrapper type:

```typescript
ExampleMessage.encode({ name: 'foo' }) // => { name: { value: 'foo' } }, in binary
ExampleMessage.encode({ name: 'foo' }); // => { name: { value: 'foo' } }, in binary
```

When calling toJSON, the value is not converted, because wrapper types are idiomatic in JSON.

```typescript
ExampleMessage.toJSON({ name: 'foo' }) // => { name: 'foo' }
ExampleMessage.toJSON({ name: 'foo' }); // => { name: 'foo' }
```

## JSON Types (Struct Types)
Expand All @@ -687,7 +697,7 @@ For this reason, Protobuf offers several additional types to represent arbitrary

These are called Struct Types, and can be imported in `.proto` files with `import "google/protobuf/struct.proto"`.

- [google.protobuf.Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#Value) &lrarr; `any`
- [google.protobuf.Value](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#Value) &lrarr; `any`
- This is the most general type, and can represent any JSON value (i.e. `number | string | boolean | null | array | object`).
- [google.protobuf.ListValue](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#ListValue) &lrarr; `any[]`
- To represent a JSON array
Expand Down Expand Up @@ -717,8 +727,9 @@ interface ExampleMessage {
```

Encoding a JSON value embedded in a message, converts it to a Struct Type:

```typescript
ExampleMessage.encode({ anything: { "name": "hello" } })
ExampleMessage.encode({ anything: { name: 'hello' } });
/* Outputs the following structure, encoded in protobuf binary format:
{
anything: Value {
Expand All @@ -735,7 +746,7 @@ ExampleMessage.encode({ anything: { "name": "hello" } })
}
}*/

ExampleMessage.encode({ anything: true })
ExampleMessage.encode({ anything: true });
/* Outputs the following structure encoded in protobuf binary format:
{
anything: Value {
Expand Down
Loading