From aab957b1e3bc0a7465ab068a4659e7aaf392e775 Mon Sep 17 00:00:00 2001 From: Stephanie DiBenedetto Date: Fri, 22 Jul 2022 16:24:12 -0700 Subject: [PATCH 1/8] Add generated code docs as docs/index.md --- docs/index.md | 388 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 388 insertions(+) create mode 100644 docs/index.md diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..8a80a9f --- /dev/null +++ b/docs/index.md @@ -0,0 +1,388 @@ + +# JavaScript Generated Code + +This page describes exactly what JavaScript code the protocol buffer compiler +generates for any given protocol definition. Any differences between proto2 and +proto3 generated code are highlighted. You should read the +[proto2 language guide](https://developers.google.com/protocol-buffers/docs/proto) and/or the +[proto3 language guide](https://developers.google.com/protocol-buffers/docs/proto3) before reading this document. + +## Compiler Invocation +The protocol buffer compiler produces JavaScript output when invoked with the +`--js_out=` command-line flag. The parameter to the `--js_out=` option is the +directory where you want the compiler to write your JavaScript output. The exact +output depends on whether you want to use Closure-style imports or +CommonJS-style imports; the compiler supports both. + +**Note:** Support for ES6-style imports is not implemented yet. Browsers can be supported by using Browserify, webpack, Closure Compiler, or similar to resolve imports at compile time. + +### Closure Imports +By default, the compiler generates code with Closure-style imports. If you +specify a `library` option when running the compiler, the compiler creates a +single `.js` file with your specified library name. Otherwise the compiler +generates a `.js` file for each *message* in your `.proto` file. The names of +the output files are computed by taking the `library` value or message name +(lowercased), with the following changes. + +So, for example, let's say you invoke the compiler as follows: + +``` +protoc --proto_path=src --js_out=library=whizz/ponycopter,binary:build/gen src/foo.proto src/bar/baz.proto +``` + +The compiler will read the files `src/foo.proto` and `src/bar/baz.proto` and +produce a single output file, `build/gen/whizz/ponycopter.js`. The compiler will +automatically create the directory `build/gen/whizz` if necessary, but it will +*not* create `build` or `build/gen`; they must already exist. +The generated file(s) `goog.provide()` all 'the types defined in your `.proto` +file(s), and `goog.require()` many types in the core protocol buffers library +and Google Closure library. Make sure that your `goog.provide()` / +`goog.require()` setup can find all of your generated code, +[the core library `.js` files](https://github.com/protocolbuffers/protobuf-javascript) +and the Google Closure library itself. + +You should be able to import your generated types with statements like: + +```js +goog.require('proto.my.package.MyMessage'); +var message = proto.my.package.MyMessage(); +``` + +### CommonJS Imports +To specify that you want to use CommonJS-style imports instead of the default +Closure style, you run the compiler with the `import_style=commonjs` option. The +names of the output files are computed by taking the name of the each input +`.proto` file and making two changes. + +**Note:** Specifying a `library` option is ignored with this import style. + +So, for example, let's say you invoke the compiler as follows: + +``` +protoc --proto_path=src --js_out=import_style=commonjs,binary:build/gen src/foo.proto src/bar/baz.proto +``` + +The compiler will read the files `src/foo.proto` and `src/bar/baz.proto` and +produce two output files: `build/gen/foo_pb.js` and `build/gen/bar/baz_pb.js`. +The compiler will automatically create the directory `build/gen/bar` if +necessary, but it will *not* create `build` or `build/gen`; they must already +exist. + +The generated code depends on the core runtime, which should be in a file +called `google-protobuf.js`. If you installed protoc via `npm`, this file should +already be built and available. If you are running from GitHub, you need to +build it first by running: + +``` +PROTOC=/path/to/protoc PROTOC_INC=/path/to/proto/include gulp dist +``` + +You should be able to import your generated types with statements like: + +```js +var messages = require('./messages_pb'); +var message = new messages.MyMessage(); +``` + +### Compiler Options +The protocol buffer compiler for JavaScript has many options to customize its +output in addition to the `library` and `import_style` options mentioned above. +For example: + +* `binary`: Using this option generates code that lets you serialize and deserialize your proto from the protocol buffers binary wire format. We recommend that you enable this option. + `--js_out=library=myprotos_lib.js,binary:.` + +As in the above examples, multiple options can be specified, separated by commas. You can see a complete list of available options in [js_generator.h](https://github.com/protocolbuffers/protobuf-javascript/blob/59a828fc713538404dcc9de8f42b4abfcfa5eb7d/generator/js_generator.h#L62-L76). + +## Packages +### Packages and Closure Imports +If you are using Closure-style imports and a `.proto` file contains a package +declaration, the generated code uses the proto's `package` as part of the +JavaScript namespace for your message types. For example, a proto package name +of `example.high_score` results in a JavaScript namespace of `proto.example.high_score`. + +```js +goog.provide('proto.example.high_score.Ponycopter'); +``` + +Otherwise, if a `.proto` file does not contain a package declaration, the +generated code just uses `proto` as the namespace for your message types, which +is the root of the protocol buffers namespace. + +### Packages and CommonJS Imports +If you are using CommonJS-style imports, any package declarations in your +`.proto` files are ignored by the compiler. + +## Messages +Given a simple message declaration: + +```protobuf +message Foo {} +``` + +the protocol buffer compiler generates a class called `Foo`. `Foo` inherits +from [`jspb.Message`](https://github.com/protocolbuffers/protobuf-javascript/blob/59a828fc713538404dcc9de8f42b4abfcfa5eb7d/message.js). + +You should *not* create your own `Foo` subclasses. Generated classes are not +designed for subclassing and may lead to "fragile base class" problems. + +Your generated class has accessors for all its fields (which we'll look at in +the following sections) and the following methods that apply to the entire +message: + +* `toObject()`: Returns an object representation of the message, suitable for use in Soy templates. This method comes in static and instance versions. Field names that are [reserved in JavaScript]("http://www.w3schools.com/js/js_reserved.asp") are renamed to `pb_name`. If you don't want to generate this method (for instance, if you're not going to use it and are concerned about code size), set [jspb.Message.GENERATE_TO_OBJECT](https://github.com/protocolbuffers/protobuf-javascript/blob/59a828fc713538404dcc9de8f42b4abfcfa5eb7d/message.js#L174) to false before code generation. Note that this representation is not the same as [proto3's JSON representation](https://developers.google.com/protocol-buffers/docs/proto3#json). +* `clone()`: Creates a deep clone of this message and its fields. + +The following methods are also provided if you have enabled the `binary` +option when generating your code: + +* `deserializeBinary()`: Static method. Deserializes a message from protocol buffers binary wire format and returns a new populated message object. Does not preserve any unknown fields in the binary message. +* `deserializeBinaryFromReader()`: Static method. Deserializes a message in protocol buffers binary wire format from the provided [BinaryReader](https://github.com/protocolbuffers/protobuf-javascript/blob/59a828fc713538404dcc9de8f42b4abfcfa5eb7d/binary/reader.js) into the provided message object. Does not preserve any unknown fields in the binary message. +* `serializeBinary()`: Serializes this message to protocol buffers binary wire format. +* `serializeBinaryToWriter()`: Serializes this message in protocol buffers binary wire format to the specified [BinaryWriter](https://github.com/protocolbuffers/protobuf-javascript/blob/59a828fc713538404dcc9de8f42b4abfcfa5eb7d/binary/writer.js). This method has a static variant where you can serialize a specified message to the BinaryWriter. + + +## Fields +The protocol buffer compiler generates accessors for each field in your +protocol buffer message. The exact accessors depend on its type and whether it +is a singular, repeated, map, or oneof field. + +Note that the generated accessors always use camel-case naming, even if the +field name in the `.proto` file uses lower-case with underscores +([as it should](https://developers.google.com/protocol-buffers/docs/style)). The case-conversion works as +follows: + +The proto field `foo_bar_baz` has, for example, a `getFooBarBaz()` method. + + +### Singular Scalar Fields (proto2) +For either of these field definitions: + +```protobuf +optional int32 foo = 1; +required int32 foo = 1; +``` + +the compiler generates the following instance methods: + +* `setFoo()`: Set the value of `foo`. +* `getFoo()`: Get the value of `foo`. If the field has not been set, returns the default value for its type. +* `hasFoo()`: Returns `true` if this field has been set. +* `clearFoo()`: Clears the value of this field: after this has been called `hasFoo()` returns `false` and `getFoo()` returns the default value. + +Similar methods are generated for any of protocol buffers' +[scalar types](https://developers.google.com/protocol-buffers/docs/proto#scalar). + +### Singular Scalar Fields (proto3) +For this field definition: + +```protobuf +int32 foo = 1; +``` + +the compiler generates the following instance methods: + +* `setFoo()`: Set the value of `foo`. +* `getFoo()`: Get the value of `foo`. + + +Similar methods are generated for any of protocol buffers' +[scalar types](https://developers.google.com/protocol-buffers/docs/proto3). + + +### Bytes Fields +For this field definition: + +```protobuf +bytes foo = 1; +``` + +the compiler generates the same methods as for other scalar value types. The +`set..` method accepts either a base-64 encoded string or a `Uint8Array`. The +`get..` method returns whichever representation was set last. However, there are +also special methods generated that allow you to coerce the returned +representation to your preferred version: + +* `getFoo_asB64()`: Returns the value of `foo` as a base-64 encoded string. +* `getFoo_asU8()`: Returns the value of `foo` as a `Uint8Array`. + +### Singular Message Fields +Given the message type: + +```protobuf +message Bar {} +``` + +For a message with a `Bar` field: + +```protobuf +// proto2 +message Baz { + optional Bar foo = 1; + // The generated code is the same result if required instead of optional. +} + +// proto3 +message Baz { + Bar foo = 1; +} +``` + +the compiler generates the following instance methods: + +* `setFoo()`: Set the value of `foo`. When called with `undefined`, it is equivalent to calling `clearFoo()`. +* `getFoo()`: Get the value of `foo`. Returns `undefined` if the field has not been set. +* `hasFoo()`: Returns `true` if this field has been set. Equivalent to `!!getFoo()`. +* `clearFoo()`: Clears the value of this field to `undefined`. + +### Repeated Fields +For this message with a repeated field: + +```protobuf +message Baz { + repeated int32 foo = 1; +} +``` + +the compiler generates the following instance methods: + +* `setFooList()`: Set the value of `foo` to the specified JavaScript array. Returns the message itself for chaining. +* `addFoo()`: Appends a value of `foo` to the end of the list of foos that was in the message. Returns the outer message for chaining **only if** the added value was a primitive. For added messages, returns the message that was added. +* `getFooList()`: Gets the value of `foo` as a JavaScript array. The returned array is never `undefined` and each element is never `undefined`. You should **no** mutate the list returned from this method. +* `clearFooList()`: Clears the value of this field to `[]`. + + +### Map Fields +For this message with a map field: + +```protobuf +message Bar {} + +message Baz { + map foo = 1; +} +``` + +the compiler generates the following instance method: + +* `getFooMap()`: Returns the [Map](https://github.com/protocolbuffers/protobuf-javascript/blob/59a828fc713538404dcc9de8f42b4abfcfa5eb7d/map.js) containing `foo`'s key-value pairs. You can then use `Map` methods to interact with the map. + +### Oneof Fields +For this message with a oneof field: + +```protobuf +package account; + +message Profile { + oneof avatar { + string image_url = 1; + bytes image_data = 2; + } +} +``` + +The class corresponding to `Profile` will have accessor methods just like regular fields (`getImageUrl()`, `getImageData()`). However, unlike regular fields, at most one of the fields in a oneof can be set at a time, so setting one field will clear the others. Also note that if you are using proto3, the compiler generates `has..` and `clear..` accessors for oneof fields, even for scalar types. + +In addition to the regular accessor methods, the compiler generates a special method to check which field in the oneof is set: for our example, the method is `getAvatarCase()`. The possible return values for this are defined in the `AvatarCase` enum: + +```js +proto.account.Profile.AvatarCase = { + AVATAR_NOT_SET: 0, + IMAGE_URL: 1, + IMAGE_DATA: 2 +}; +``` + +## Enumerations +Given an enumeration like: + +```protobuf +message SearchRequest { + enum Corpus { + UNIVERSAL = 0; + WEB = 1; + IMAGES = 2; + LOCAL = 3; + NEWS = 4; + PRODUCTS = 5; + VIDEO = 6; + } + Corpus corpus = 1; + ... +} +``` + +the protocol buffer compiler generates a corresponding JavaScript enum. + +```js +proto.SearchRequest.Corpus = { + UNIVERSAL: 0, + WEB: 1, + IMAGES: 2, + LOCAL: 3, + NEWS: 4, + PRODUCTS: 5, + VIDEO: 6 +}; +``` + +The compiler also generates getters and setters for enum fields, just like +regular singular scalar fields. Note that in proto3, you can set an enum field +to any value. In proto2, you should provide one of the specified enum values. + +## Any +Given an [`Any`](/protocol-buffers/docs/proto3#any) field like this: + +```protobuf +import "google/protobuf/any.proto"; + +package foo; + +message Bar {} + +message ErrorStatus { + string message = 1; + google.protobuf.Any details = 2; +} +``` + +In our generated code, the getter for the `details` field returns an instance of `proto.google.protobuf.Any`. This provides the following special methods: + +```js +/** + * Returns the fully qualified proto name of the packed message, if any. + * @return {string|undefined} + */ +proto.google.protobuf.Any.prototype.getTypeName; +/** + * Packs the given message instance into this Any. + * @param {!Uint8Array} serialized The serialized data to pack. + * @param {string} name The fully qualified proto name of the packed message. + * @param {string=} opt_typeUrlPrefix the type URL prefix. + */ +proto.google.protobuf.Any.prototype.pack; +/** + * @template T + * Unpacks this Any into the given message object. + * @param {function(Uint8Array):T} deserialize Function that will deserialize + * the binary data properly. + * @param {string} name The expected type name of this message object. + * @return {?T} If the name matched the expected name, returns the deserialized + * object, otherwise returns null. + */ +proto.google.protobuf.Any.prototype.unpack; +``` + +Example: + +```js +// Storing an arbitrary message type in Any. +const status = new proto.foo.ErrorStatus(); +const any = new Any(); +const binarySerialized = ...; +any.pack(binarySerialized, 'foo.Bar'); +console.log(any.getTypeName()); // foo.Bar +// Reading an arbitrary message from Any. +const bar = any.unpack(proto.foo.Bar.deserializeBinary, 'foo.Bar'); +``` + From 0bf239daa104b16507ca576fdcfaf578c8389a3a Mon Sep 17 00:00:00 2001 From: Paul Johnston Date: Sun, 1 May 2022 18:13:34 -0600 Subject: [PATCH 2/8] buildozer 'add visibility //visibility:public' //generator:protoc-gen-js --- generator/BUILD | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/generator/BUILD b/generator/BUILD index 18b6869..de1fc18 100644 --- a/generator/BUILD +++ b/generator/BUILD @@ -1,13 +1,13 @@ - cc_binary( name = "protoc-gen-js", srcs = [ - "protoc-gen-js.cc", "js_generator.cc", "js_generator.h", + "protoc-gen-js.cc", "well_known_types_embed.cc", "well_known_types_embed.h", ], + visibility = ["//visibility:public"], deps = [ #"@com_google_absl//absl/base:core_headers", #"@com_google_absl//absl/container:flat_hash_map", From 8972d9a8088353e8fd2207531cb7b2cb14657e23 Mon Sep 17 00:00:00 2001 From: David Weitzman Date: Sun, 5 Jun 2022 12:48:26 -0700 Subject: [PATCH 3/8] Catch non-strings in BinaryEncoder.writeString The javascript serializeBinary() function silently tosses away any non-string placed into what should be a string field, writing the equivalent of an empty string instead Not sure if or why anyone would be relying on that existing arbitrary behavior today, but silent data loss seems very undesirable and worth defending against --- binary/decoder_test.js | 10 ++++++++++ binary/encoder.js | 3 +++ 2 files changed, 13 insertions(+) diff --git a/binary/decoder_test.js b/binary/decoder_test.js index 0fddc4b..fc0384b 100644 --- a/binary/decoder_test.js +++ b/binary/decoder_test.js @@ -376,6 +376,16 @@ describe('binaryDecoderTest', function() { assertEquals(utf8_four_bytes, decoder.readString(utf8_four_bytes.length)); }); + /** + * Verifies that passing a non-string to writeString raises an error. + */ + it('testBadString', function() { + var encoder = new jspb.BinaryEncoder(); + + assertThrows(function() {encoder.writeString(42)}); + assertThrows(function() {encoder.writeString(null)}); + }); + /** * Verifies that misuse of the decoder class triggers assertions. */ diff --git a/binary/encoder.js b/binary/encoder.js index 726481b..2c59a97 100644 --- a/binary/encoder.js +++ b/binary/encoder.js @@ -473,6 +473,9 @@ jspb.BinaryEncoder.prototype.writeFixedHash64 = function(hash) { jspb.BinaryEncoder.prototype.writeString = function(value) { var oldLength = this.buffer_.length; + // Protect against non-string values being silently ignored. + goog.asserts.assert(value.charCodeAt); + for (var i = 0; i < value.length; i++) { var c = value.charCodeAt(i); From 48bd68ea70c113f9088e07b19d012aabadbd9b61 Mon Sep 17 00:00:00 2001 From: David Weitzman Date: Fri, 22 Jul 2022 11:37:04 -0700 Subject: [PATCH 4/8] Use goog.asserts.assertString --- binary/encoder.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/binary/encoder.js b/binary/encoder.js index 2c59a97..de08748 100644 --- a/binary/encoder.js +++ b/binary/encoder.js @@ -474,7 +474,7 @@ jspb.BinaryEncoder.prototype.writeString = function(value) { var oldLength = this.buffer_.length; // Protect against non-string values being silently ignored. - goog.asserts.assert(value.charCodeAt); + goog.asserts.assertString(value); for (var i = 0; i < value.length; i++) { From a3f086825e5b3c4df97bbde4d685f0ac013791a7 Mon Sep 17 00:00:00 2001 From: Paul Grau Date: Tue, 17 May 2022 08:46:31 +0900 Subject: [PATCH 5/8] Fix uint8array check --- binary/utils.js | 14 +++++++++----- binary/utils_test.js | 4 +++- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/binary/utils.js b/binary/utils.js index d281027..d912305 100644 --- a/binary/utils.js +++ b/binary/utils.js @@ -1051,11 +1051,6 @@ jspb.utils.byteSourceToUint8Array = function(data) { return /** @type {!Uint8Array} */(new Uint8Array(data)); } - if (typeof Buffer != 'undefined' && data.constructor === Buffer) { - return /** @type {!Uint8Array} */ ( - new Uint8Array(/** @type {?} */ (data))); - } - if (data.constructor === Array) { data = /** @type {!Array} */(data); return /** @type {!Uint8Array} */(new Uint8Array(data)); @@ -1066,6 +1061,15 @@ jspb.utils.byteSourceToUint8Array = function(data) { return goog.crypt.base64.decodeStringToUint8Array(data); } + if (data instanceof Uint8Array) { + // Support types like nodejs Buffer and other subclasses of Uint8Array. + data = /** @type {!Uint8Array} */ (data); + // Make a shallow copy to ensure we only ever deal with Uint8Array + // exactly to ensure monomorphism. + return /** @type {!Uint8Array} */ ( + new Uint8Array(data.buffer, data.byteOffset, data.byteLength)); + } + goog.asserts.fail('Type not convertible to Uint8Array.'); return /** @type {!Uint8Array} */(new Uint8Array(0)); }; diff --git a/binary/utils_test.js b/binary/utils_test.js index 70585db..d1e4b12 100644 --- a/binary/utils_test.js +++ b/binary/utils_test.js @@ -719,7 +719,6 @@ describe('binaryUtilsTest', function() { var sourceBytes = new Uint8Array(sourceData); var sourceBuffer = sourceBytes.buffer; var sourceBase64 = goog.crypt.base64.encodeByteArray(sourceData); - var sourceString = goog.crypt.byteArrayToString(sourceData); function check(result) { expect(result.constructor).toEqual(Uint8Array); @@ -740,5 +739,8 @@ describe('binaryUtilsTest', function() { // Converting base64-encoded strings into Uint8Arrays should work. check(convert(sourceBase64)); + + // Existing Uint8Array into Uint8Arrays should work. + check(convert(sourceUint8Array)); }); }); From 0f35d29d6aa81512c4770803678ce64cb0d98709 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Noel=20Kim=20=28=EA=B9=80=EB=AF=BC=ED=98=81=29?= Date: Fri, 29 Apr 2022 14:06:53 +0900 Subject: [PATCH 6/8] chore(typo): add line break in clear map method --- generator/js_generator.cc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/generator/js_generator.cc b/generator/js_generator.cc index eb8aa73..446c29d 100644 --- a/generator/js_generator.cc +++ b/generator/js_generator.cc @@ -2818,7 +2818,7 @@ void Generator::GenerateClassField(const GeneratorOptions& options, " */\n" "$class$.prototype.$clearername$ = function() {\n" " this.$gettername$().clear();\n" - " return this;" + " return this;\n" "};\n" "\n" "\n", From 1c001d3862ac90ef3f7c73b47ecb81475b21c7ab Mon Sep 17 00:00:00 2001 From: Stephanie DiBenedetto Date: Fri, 22 Jul 2022 16:44:36 -0700 Subject: [PATCH 7/8] Remove undefined var usage in uint8arrays conversion test --- binary/utils_test.js | 3 --- 1 file changed, 3 deletions(-) diff --git a/binary/utils_test.js b/binary/utils_test.js index d1e4b12..995f129 100644 --- a/binary/utils_test.js +++ b/binary/utils_test.js @@ -739,8 +739,5 @@ describe('binaryUtilsTest', function() { // Converting base64-encoded strings into Uint8Arrays should work. check(convert(sourceBase64)); - - // Existing Uint8Array into Uint8Arrays should work. - check(convert(sourceUint8Array)); }); }); From bf0999ec27331bd6cd1ce54ee309b59da04a1dcb Mon Sep 17 00:00:00 2001 From: Stephanie DiBenedetto Date: Fri, 22 Jul 2022 16:24:12 -0700 Subject: [PATCH 8/8] Add generated code docs as docs/index.md --- docs/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/index.md b/docs/index.md index 8a80a9f..704ffc5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -27,7 +27,7 @@ the output files are computed by taking the `library` value or message name So, for example, let's say you invoke the compiler as follows: ``` -protoc --proto_path=src --js_out=library=whizz/ponycopter,binary:build/gen src/foo.proto src/bar/baz.proto +protoc --plugin=protoc-gen-js=/path/to/protobuf-javascript/bazel-bin/generator/protoc-gen-js --proto_path=src --js_out=library=whizz/ponycopter,binary:build/gen src/foo.proto src/bar/baz.proto ``` The compiler will read the files `src/foo.proto` and `src/bar/baz.proto` and @@ -45,7 +45,7 @@ You should be able to import your generated types with statements like: ```js goog.require('proto.my.package.MyMessage'); -var message = proto.my.package.MyMessage(); +const message = proto.my.package.MyMessage(); ``` ### CommonJS Imports @@ -59,7 +59,7 @@ names of the output files are computed by taking the name of the each input So, for example, let's say you invoke the compiler as follows: ``` -protoc --proto_path=src --js_out=import_style=commonjs,binary:build/gen src/foo.proto src/bar/baz.proto +protoc --plugin=protoc-gen-js=/path/to/protobuf-javascript/bazel-bin/generator/protoc-gen-js --proto_path=src --js_out=import_style=commonjs,binary:build/gen src/foo.proto src/bar/baz.proto ``` The compiler will read the files `src/foo.proto` and `src/bar/baz.proto` and @@ -80,8 +80,8 @@ PROTOC=/path/to/protoc PROTOC_INC=/path/to/proto/include gulp dist You should be able to import your generated types with statements like: ```js -var messages = require('./messages_pb'); -var message = new messages.MyMessage(); +const messages = require('./messages_pb'); +const message = new messages.MyMessage(); ``` ### Compiler Options