Add support for GELU and approximate activation functions #8224

Vectorrent · 2024-03-27T06:13:50Z

I added a basic implementation of the gelu and gelu_new activation functions, which can be called via the appropriate string names accepted by built-in layers. Tests are included. Docker configs were used for testing in an isolated environment.

I considered passing an approximate = true argument to the apply() method, in trying to be consistent with Keras - but it wasn't clear to me how that should work. TypeScript would not allow me to extend the Gelu class to an approximate version, and since the string references need to refer to a static class name directly... it just seemed easier to create two separate classes here.

I hope you'll accept this PR, since it would be very useful to me personally (I'm building transformers). Please let me know if there's anything we should change before merging!

google-cla · 2024-03-27T06:13:54Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

pyu10055 · 2024-04-04T18:15:33Z

/gcbrun

pyu10055 · 2024-04-04T18:24:12Z

Thanks for the contribution, I am not sure why you cannot add a parameter (approximation=true) to the apply function similar to Swish

apply(x: Tensor, alpha = 1): Tensor {
    return tidy(() => tfc.mul(tfc.sigmoid(tfc.mul(x, alpha)), x));
  }

can you explain?

pyu10055 · 2024-04-04T18:30:41Z

/gcbrun

Vectorrent · 2024-04-04T18:39:28Z

can you explain?

Thanks for the followup! So, in my original attempt, I struggled to integrate an approximate=true flag because of the way these string arguments work:

export const activationOptions = stringLiteralArray([
  'elu', 'hard_sigmoid', 'linear', 'relu', 'relu6', 'selu', 'sigmoid',
  'softmax', 'softplus', 'softsign', 'tanh', 'swish', 'mish', 'gelu', 'gelu_new'
]);

Because of the way these string arguments must match an activation's class name... I wasn't sure how to point two string args to the same class, with a different approximate flag toggled. I tried to extend GeluNew from Gelu, but TypeScript wouldn't allow me to do that. Not knowing what else to try, I just submitted it as-is.

I could maybe look into this a bit more later.

pyu10055 · 2024-04-05T00:12:15Z

can you explain?

Thanks for the followup! So, in my original attempt, I struggled to integrate an approximate=true flag because of the way these string arguments work:
export const activationOptions = stringLiteralArray([
  'elu', 'hard_sigmoid', 'linear', 'relu', 'relu6', 'selu', 'sigmoid',
  'softmax', 'softplus', 'softsign', 'tanh', 'swish', 'mish', 'gelu', 'gelu_new'
]);
Because of the way these string arguments must match an activation's class name... I wasn't sure how to point two string args to the same class, with a different approximate flag toggled. I tried to extend GeluNew from Gelu, but TypeScript wouldn't allow me to do that. Not knowing what else to try, I just submitted it as-is.

I could maybe look into this a bit more later.

ok, these are two different activations, are you referring sharing the implementation? It is ok to have two different classes.

pyu10055 · 2024-04-05T00:12:23Z

/gcbrun

Vectorrent · 2024-04-05T00:15:33Z

ok, these are two different activations, are you referring sharing the implementation? It is ok to have two different classes.

Yes, I was going to share the implementation, but TypeScript didn't like that. Thanks for the followup, and please let me know if you need anything else from me!

Dockerfile

docker-compose.yml

pyu10055 · 2024-04-05T16:35:58Z

tfjs-layers/src/activations_test.ts

+  });
+});
+
+describeMathCPUAndGPU('gelu_new activation', () => {


this test is failing on webgl

https://btx.cloud.google.com/invocations/e17bf0ae-be69-4c2c-ae42-898362837087/targets/%2F%2Ftfjs-layers:bs_chrome_mac_tfjs-layers_webgl2_test;config=558edca9061547b5089f0232d6a9290fa00acb52e12d7215b154805a33d24105/log

I can't see the error, so I don't know what's failing. I can't imagine why webgl would fail while other backends would succeed, but maybe it's a difference in floating point precision between environments?

Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 1D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:378:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29642:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 1D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:378:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29642:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 1D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:378:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29642:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 113 of 4308 (1 FAILED) (0 secs / 2.725 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 1D all equal FAILED Error: Arrays differ: actual[0] = -0.15865525603294373, expected[0] = 0.15880799293518066. Actual: -0.15865525603294373,-0.15865525603294373,-0.15865525603294373,-0.15865525603294373. Expected: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:383:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29647:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 1D all equal FAILED Error: Arrays differ: actual[0] = -0.15865525603294373, expected[0] = 0.15880799293518066. Actual: -0.15865525603294373,-0.15865525603294373,-0.15865525603294373,-0.15865525603294373. Expected: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:383:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29647:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 1D all equal FAILED Error: Arrays differ: actual[0] = -0.15865525603294373, expected[0] = 0.15880799293518066. Actual: -0.15865525603294373,-0.15865525603294373,-0.15865525603294373,-0.15865525603294373. Expected: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:383:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29647:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 114 of 4308 (2 FAILED) (0 secs / 2.726 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 2D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:389:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29652:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 2D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:389:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29652:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 2D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:389:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29652:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 115 of 4308 (3 FAILED) (0 secs / 2.727 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 3D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:395:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29657:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 3D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:395:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29657:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation cpu {} 3D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:395:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29657:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 116 of 4308 (4 FAILED) (0 secs / 2.728 secs) WARN: 'Spec 'gelu activation cpu {} Does not leak' has no expectations.' WARN: 'Spec 'gelu activation cpu {} Does not leak' has no expectations.' WARN: 'Spec 'gelu activation cpu {} Does not leak' has no expectations.' Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 116 of 4308 (4 FAILED) (0 secs / 2.728 secs) Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 117 of 4308 (4 FAILED) (0 secs / 2.729 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:378:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29642:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:378:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29642:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:378:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29642:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 118 of 4308 (5 FAILED) (0 secs / 2.882 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D all equal FAILED Error: Arrays differ: actual[0] = -0.15865522623062134, expected[0] = 0.15880799293518066. Actual: -0.15865522623062134,-0.15865522623062134,-0.15865522623062134,-0.15865522623062134. Expected: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:383:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29647:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D all equal FAILED Error: Arrays differ: actual[0] = -0.15865522623062134, expected[0] = 0.15880799293518066. Actual: -0.15865522623062134,-0.15865522623062134,-0.15865522623062134,-0.15865522623062134. Expected: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:383:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29647:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D all equal FAILED Error: Arrays differ: actual[0] = -0.15865522623062134, expected[0] = 0.15880799293518066. Actual: -0.15865522623062134,-0.15865522623062134,-0.15865522623062134,-0.15865522623062134. Expected: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:383:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29647:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 119 of 4308 (6 FAILED) (0 secs / 2.885 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 2D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9,0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:389:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29652:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 2D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9,0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:389:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29652:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 2D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9,0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:389:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29652:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 120 of 4308 (7 FAILED) (0 secs / 2.974 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 3D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9,0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:395:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29657:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 3D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9,0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:395:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29657:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 3D FAILED Error: Arrays differ: actual[0] = 0, expected[0] = 0.5. Actual: 0,0.8413447737693787,2.995950222015381,9,0,0.8413447737693787,2.995950222015381,9. Expected: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:395:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29657:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 121 of 4308 (8 FAILED) (0 secs / 3.034 secs) WARN: 'Spec 'gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} Does not leak' has no expectations.' WARN: 'Spec 'gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} Does not leak' has no expectations.' WARN: 'Spec 'gelu activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} Does not leak' has no expectations.' Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 121 of 4308 (8 FAILED) (0 secs / 3.034 secs) Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 122 of 4308 (8 FAILED) (0 secs / 3.035 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 1D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:411:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29669:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 1D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:411:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29669:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 1D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:411:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29669:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 123 of 4308 (9 FAILED) (0 secs / 3.036 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 1D all equal FAILED Error: Arrays differ: actual[0] = 0.15880802273750305, expected[0] = -0.1586553007364273. Actual: 0.15880802273750305,0.15880802273750305,0.15880802273750305,0.15880802273750305. Expected: -0.1586553007364273,-0.1586553007364273,-0.1586553007364273,-0.1586553007364273. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:417:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29674:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 1D all equal FAILED Error: Arrays differ: actual[0] = 0.15880802273750305, expected[0] = -0.1586553007364273. Actual: 0.15880802273750305,0.15880802273750305,0.15880802273750305,0.15880802273750305. Expected: -0.1586553007364273,-0.1586553007364273,-0.1586553007364273,-0.1586553007364273. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:417:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29674:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 1D all equal FAILED Error: Arrays differ: actual[0] = 0.15880802273750305, expected[0] = -0.1586553007364273. Actual: 0.15880802273750305,0.15880802273750305,0.15880802273750305,0.15880802273750305. Expected: -0.1586553007364273,-0.1586553007364273,-0.1586553007364273,-0.1586553007364273. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:417:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29674:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 124 of 4308 (10 FAILED) (0 secs / 3.037 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 2D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.5611233115196228,2.850890874862671,8.998000144958496,0,0.5611233115196228,2.850890874862671,8.998000144958496. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:424:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29682:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 2D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.5611233115196228,2.850890874862671,8.998000144958496,0,0.5611233115196228,2.850890874862671,8.998000144958496. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:424:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29682:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 2D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.5611233115196228,2.850890874862671,8.998000144958496,0,0.5611233115196228,2.850890874862671,8.998000144958496. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:424:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29682:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 125 of 4308 (11 FAILED) (0 secs / 3.038 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 3D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:430:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29687:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 3D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:430:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29687:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation cpu {} 3D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.998787522315979,1,0.5,0.8411920070648193,0.998787522315979,1. Expected: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:430:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29687:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 126 of 4308 (12 FAILED) (0 secs / 3.039 secs) WARN: 'Spec 'gelu_new activation cpu {} Does not leak' has no expectations.' WARN: 'Spec 'gelu_new activation cpu {} Does not leak' has no expectations.' WARN: 'Spec 'gelu_new activation cpu {} Does not leak' has no expectations.' Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 126 of 4308 (12 FAILED) (0 secs / 3.039 secs) Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 127 of 4308 (12 FAILED) (0 secs / 3.04 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:411:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29669:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:411:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29669:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:411:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29669:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 128 of 4308 (13 FAILED) (0 secs / 3.176 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D all equal FAILED Error: Arrays differ: actual[0] = 0.15880799293518066, expected[0] = -0.1586553007364273. Actual: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. Expected: -0.1586553007364273,-0.1586553007364273,-0.1586553007364273,-0.1586553007364273. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:417:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29674:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D all equal FAILED Error: Arrays differ: actual[0] = 0.15880799293518066, expected[0] = -0.1586553007364273. Actual: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. Expected: -0.1586553007364273,-0.1586553007364273,-0.1586553007364273,-0.1586553007364273. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:417:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29674:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D all equal FAILED Error: Arrays differ: actual[0] = 0.15880799293518066, expected[0] = -0.1586553007364273. Actual: 0.15880799293518066,0.15880799293518066,0.15880799293518066,0.15880799293518066. Expected: -0.1586553007364273,-0.1586553007364273,-0.1586553007364273,-0.1586553007364273. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:417:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29674:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 129 of 4308 (14 FAILED) (0 secs / 3.181 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 2D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1,0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.5611233115196228,2.850890874862671,8.998000144958496,0,0.5611233115196228,2.850890874862671,8.998000144958496. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:424:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29682:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 2D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1,0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.5611233115196228,2.850890874862671,8.998000144958496,0,0.5611233115196228,2.850890874862671,8.998000144958496. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:424:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29682:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 2D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1,0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.5611233115196228,2.850890874862671,8.998000144958496,0,0.5611233115196228,2.850890874862671,8.998000144958496. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:424:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29682:11) at Chrome 116.0.0.0 (Mac OS 10.15.7): Executed 130 of 4308 (15 FAILED) (0 secs / 3.288 secs) Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 3D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1,0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:430:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29687:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 3D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1,0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:430:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29687:11) at Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 3D FAILED Error: Arrays differ: actual[0] = 0.5, expected[0] = 0. Actual: 0.5,0.8411920070648193,0.9987875819206238,1,0.5,0.8411920070648193,0.9987875819206238,1. Expected: 0,0.8413447141647339,2.995950222015381,9,0,0.8413447141647339,2.995950222015381,9. at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15) at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12) at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23) at UserContext. (tfjs-layers/src/activations_test.ts:430:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29687:11) at

I'm not sure how I screwed up so badly, but every single test value was wrong. I just went through and updated every one of them by-hand; tests should be fixed now. Sorry about that!

no worries, thanks for fixing the test.
can you run following in the tfjs directory:
tslint -p tsconfig_tslint.json
and fix the lint errors.

Done! I'll remember to check the linting in future commits.

Thanks for the PR! I'm back in office now, so I've taken a look.

I think this implementation of gelu_new might be computing just Φ(x) instead of x Φ(x)

According to help(keras.activations.gelu) in python,

Returns: The gaussian error linear activation: `0.5 * x * (1 + tanh(sqrt(2 / pi) * (x + 0.044715 * x^3)))` if `approximate` is `True` or `x * P(X <= x) = 0.5 * x * (1 + erf(x / sqrt(2)))`, where `P(X) ~ N(0, 1)`, if `approximate` is `False`.

0.5 * x * (1 + tanh(sqrt(2 / pi) * (x + 0.044715 * x^3))) is the approximate form that gelu_new is computing, but I think the implementation is missing the * x. That's why all the values were wrong.

I tried adding a * x to the implementation and the values look closer:

gelu(initX).print() [0, 0.8413447, 2.9959502, 9] geluNew(initX).print() [0, 0.841192, 2.9963627, 9]

pyu10055 · 2024-04-08T23:14:01Z

/gcbrun

pyu10055 · 2024-04-08T23:14:31Z

/gcbrun

pyu10055 · 2024-04-10T21:04:29Z

/gcbrun

pyu10055 · 2024-04-10T21:54:08Z

one more lint error

$ tslint -p tsconfig_tslint.json
/workspace/tfjs-layers/src/activations_test.ts:463:1
ERROR: 463:1  no-consecutive-blank-lines  Consecutive blank lines are forbidden

Vectorrent · 2024-04-10T21:58:04Z

Fixed.

mattsoulanille · 2024-04-11T20:07:09Z

/gcbrun

mattsoulanille · 2024-04-11T21:52:44Z

tfjs-layers/src/activations_test.ts

+  });
+});
+
+describeMathCPUAndGPU('gelu_new activation', () => {


Thanks for the PR! I'm back in office now, so I've taken a look.

I think this implementation of gelu_new might be computing just Φ(x) instead of x Φ(x)

According to help(keras.activations.gelu) in python,

Returns: The gaussian error linear activation: `0.5 * x * (1 + tanh(sqrt(2 / pi) * (x + 0.044715 * x^3)))` if `approximate` is `True` or `x * P(X <= x) = 0.5 * x * (1 + erf(x / sqrt(2)))`, where `P(X) ~ N(0, 1)`, if `approximate` is `False`.

0.5 * x * (1 + tanh(sqrt(2 / pi) * (x + 0.044715 * x^3))) is the approximate form that gelu_new is computing, but I think the implementation is missing the * x. That's why all the values were wrong.

I tried adding a * x to the implementation and the values look closer:

gelu(initX).print() [0, 0.8413447, 2.9959502, 9] geluNew(initX).print() [0, 0.841192, 2.9963627, 9]

Vectorrent · 2024-04-11T22:24:22Z

Ah, you are correct about that. I've fixed the calculation; please let me know if that still looks wrong.

pyu10055 · 2024-04-12T17:50:01Z

/rungcb

pyu10055 · 2024-04-12T17:50:19Z

/gcbrun

mattsoulanille · 2024-04-12T18:33:03Z

Looks like there's a sign error with the 1D all equal test.

Chrome 116.0.0.0 (Mac OS 10.15.7) gelu_new activation webgl2 {"WEBGL_VERSION":2,"WEBGL_CPU_FORWARD":false,"WEBGL_SIZE_UPLOAD_UNIFORM":0} 1D all equal FAILED
	Error: Arrays differ: actual[0] = -0.15880799293518066, expected[0] = 0.15880802273750305.
	Actual:   -0.15880799293518066,-0.15880799293518066,-0.15880799293518066,-0.15880799293518066.
	Expected: 0.15880802273750305,0.15880802273750305,0.15880802273750305,0.15880802273750305.
	    at expectArraysPredicate (tfjs-core/src/test_util.ts:90:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10062:15)
	    at Object.expectArraysClose (tfjs-core/src/test_util.ts:32:10 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:10024:12)
	    at expectTensorsClose (tfjs-layers/src/utils/test_utils.ts:51:13 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29245:23)
	    at UserContext. (tfjs-layers/src/activations_test.ts:439:5 <- tfjs/tfjs-layers/src/tfjs-layers_test_bundle.js:29700:11)
	    at

mattsoulanille · 2024-04-12T18:36:57Z

/gcbrun

Vectorrent · 2024-04-12T18:37:28Z

My mistake, should be fixed now.

mattsoulanille

LGTM

mattsoulanille · 2024-04-12T18:40:52Z

@pyu10055 The gelu_new test failures are fixed. Please take another look when you get a chance. Thanks!

pyu10055 · 2024-04-12T19:59:54Z

/gcbrun

Vectorrent added 2 commits March 26, 2024 10:46

add docker configs for isolated testing

f4ad2b1

implement gelu and gelu_new as separate activations

d90d735

pyu10055 added 2 commits April 4, 2024 11:28

Update activations.ts

f27de34

Update activations_test.ts

6352f8b

Update activations_test.ts

890182d

pyu10055 requested review from mattsoulanille and pyu10055 April 5, 2024 00:13

pyu10055 reviewed Apr 5, 2024

View reviewed changes

Vectorrent added 2 commits April 6, 2024 08:30

remove docker files

679a162

fix activation tests

7181b07

Merge branch 'master' into add-new-activations

f78d5d9

fix lint errors

742e19d

remove extra blank line

376ba19

mattsoulanille requested changes Apr 11, 2024

View reviewed changes

fix gelu_new calc

ed02f32

fix 1D test

bfaee9e

mattsoulanille approved these changes Apr 12, 2024

View reviewed changes

Merge branch 'master' into add-new-activations

7b38dbc

pyu10055 approved these changes Apr 12, 2024

View reviewed changes

pyu10055 merged commit c027d6a into tensorflow:master Apr 12, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for GELU and approximate activation functions #8224

Add support for GELU and approximate activation functions #8224

Vectorrent commented Mar 27, 2024 •

edited

google-cla bot commented Mar 27, 2024

pyu10055 commented Apr 4, 2024

pyu10055 commented Apr 4, 2024

pyu10055 commented Apr 4, 2024

Vectorrent commented Apr 4, 2024 •

edited

pyu10055 commented Apr 5, 2024

pyu10055 commented Apr 5, 2024

Vectorrent commented Apr 5, 2024

pyu10055 Apr 5, 2024

Vectorrent Apr 6, 2024

Vectorrent Apr 6, 2024

pyu10055 Apr 8, 2024

Vectorrent Apr 8, 2024

pyu10055 Apr 8, 2024 •

edited

Vectorrent Apr 8, 2024

mattsoulanille Apr 11, 2024

pyu10055 commented Apr 8, 2024

pyu10055 commented Apr 8, 2024

pyu10055 commented Apr 10, 2024

pyu10055 commented Apr 10, 2024

Vectorrent commented Apr 10, 2024

mattsoulanille commented Apr 11, 2024

mattsoulanille Apr 11, 2024

Vectorrent commented Apr 11, 2024

pyu10055 commented Apr 12, 2024

pyu10055 commented Apr 12, 2024

mattsoulanille commented Apr 12, 2024

mattsoulanille commented Apr 12, 2024

Vectorrent commented Apr 12, 2024

mattsoulanille left a comment

mattsoulanille commented Apr 12, 2024

pyu10055 commented Apr 12, 2024

Add support for GELU and approximate activation functions #8224

Add support for GELU and approximate activation functions #8224

Conversation

Vectorrent commented Mar 27, 2024 • edited

google-cla bot commented Mar 27, 2024

pyu10055 commented Apr 4, 2024

pyu10055 commented Apr 4, 2024

pyu10055 commented Apr 4, 2024

Vectorrent commented Apr 4, 2024 • edited

pyu10055 commented Apr 5, 2024

pyu10055 commented Apr 5, 2024

Vectorrent commented Apr 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pyu10055 Apr 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pyu10055 commented Apr 8, 2024

pyu10055 commented Apr 8, 2024

pyu10055 commented Apr 10, 2024

pyu10055 commented Apr 10, 2024

Vectorrent commented Apr 10, 2024

mattsoulanille commented Apr 11, 2024

Choose a reason for hiding this comment

Vectorrent commented Apr 11, 2024

pyu10055 commented Apr 12, 2024

pyu10055 commented Apr 12, 2024

mattsoulanille commented Apr 12, 2024

mattsoulanille commented Apr 12, 2024

Vectorrent commented Apr 12, 2024

mattsoulanille left a comment

Choose a reason for hiding this comment

mattsoulanille commented Apr 12, 2024

pyu10055 commented Apr 12, 2024

Vectorrent commented Mar 27, 2024 •

edited

Vectorrent commented Apr 4, 2024 •

edited

pyu10055 Apr 8, 2024 •

edited