Masked LM saved model testing occasionally flaking

[mattdangerw]

Our masked lm saved model testing appears to be occasionally flaking in the tf format. See https://github.com/keras-team/keras-nlp/pull/856/checks?check_run_id=12517800555 for example.

We should investigate the root cause here. This could be potentially related to the tf 2.12 release.

[chenmoneygithub]

Paste log here for easy checks:

Step #5 - "create-job": E       Mismatched value: a is different from b. 
Step #5 - "create-job": E       not close where = (array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]), array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), array([ 0,  1,  2,  3,  4,  6,  7,  8,  9, 10, 11]))
Step #5 - "create-job": E       not close lhs = [ 0.00651812 -0.03150878 -0.00669549  0.01317    -0.0128293   0.00930192
Step #5 - "create-job": E         0.02229765  0.01699986 -0.0111946   0.01243278 -0.01390067]
Step #5 - "create-job": E       not close rhs = [-0.00625689  0.03024605  0.00642716 -0.0126422   0.01231514 -0.00892914
Step #5 - "create-job": E        -0.02140406 -0.01631859  0.01074597 -0.01193453  0.01334361]
Step #5 - "create-job": E       not close dif = [0.012775   0.06175482 0.01312265 0.02581221 0.02514444 0.01823106
Step #5 - "create-job": E        0.04370171 0.03331845 0.02194057 0.02436731 0.02724428]
Step #5 - "create-job": E       not close tol = [0.01006257 0.01030246 0.01006427 0.01012642 0.01012315 0.01008929
Step #5 - "create-job": E        0.01021404 0.01016319 0.01010746 0.01011935 0.01013344]

Magically at checking time, the pos/neg is flipped. No idea what could be the issue, but seems to be a regression.