[tfjs-node] Native tensor handles leak after signed int32 tensor ID rollover

[GideonKoenig]

System information

• Have I written custom code: Yes; a minimal standalone reproduction is included below.
• OS Platform and Distribution: Debian 12, Linux x86-64.
• TensorFlow.js installed from: npm.
• TensorFlow.js version: @tensorflow/tfjs-node 4.22.0 / TFJS 4.22.0.
• Node.js version: v22.23.1.
• Backend: tensorflow CPU backend, bundled libtensorflow 2.9.1.

Describe the current behavior

tfjs-node assigns each native TFE_TensorHandle a monotonically increasing signed 32-bit ID. When the process creates its 2,147,483,648th native handle, the observed ID rolls from 2,147,483,647 to -2,147,483,648.

The JavaScript disposal path calls deleteTensor(id) only when id >= 0. Tensors created after the rollover are therefore removed from JavaScript bookkeeping, but their native handles remain in tfe_handle_map_. Native memory and getNumOfTFTensors() then grow monotonically while tf.memory().numTensors returns to baseline.

This was encountered in a real, long-running reinforcement-learning training service using tfjs-node. At approximately 22,377 handle allocations per second, the service reached the rollover after about 26.7 hours and then hit a native-memory cliff. Multiple production training runs failed on this timescale, including two controlled runs with identical configuration and seed that failed at the same training step. A captured core contained 195,793 live tensorflow::TensorHandle objects and 195,782 map IDs in the wrapped range beginning exactly at 0x80000000. The accelerated GDB reproduction below was created afterward to reproduce the same rollover without waiting more than a day.

The relevant current source appears to be:

• int32_t next_tensor_id_
• next_tensor_id_++
• the id >= 0 disposal guard

Describe the expected behavior

Disposing tensors should release their native handles regardless of how many handles the process has created previously. A long-lived process with a bounded live tensor population should not leak after a cumulative allocation threshold.

Standalone code to reproduce the issue

The natural rollover requires 2,147,483,648 handle allocations. The following GDB-assisted reproduction moves the counter thirteen IDs below INT32_MAX in a disposable Linux x86-64 process and then runs ordinary tensor operations.

reproduce.js:

'use strict';

const tf = require('@tensorflow/tfjs-node');

function snapshot(iteration) {
    console.log(JSON.stringify({
        iteration,
        logicalTensorCount: tf.memory().numTensors,
        nativeTensorCount: tf.backend().getNumOfTFTensors(),
    }));
}

snapshot(0);
for (let iteration = 1; iteration <= 250; iteration++) {
    tf.tidy(() => {
        const input = tf.tensor1d([1, 2, 3, 4]);
        const output = input.add(iteration).square().sub(input).sum();
        output.dataSync();
    });
    if (iteration <= 10 || iteration % 25 === 0) snapshot(iteration);
}
snapshot('complete');

force-counter-near-overflow.gdb for the official 4.22.0 Linux x86-64 binary:

set pagination off
set confirm off
set breakpoint pending on
set debuginfod enabled off

break tfnodejs::TFJSBackend::InsertHandle(TFE_TensorHandle*)
commands 1
    silent
    set {int}($rdi + 0x78) = 2147483635
    printf "forced TFJSBackend::next_tensor_id_ to %d\n", {int}($rdi + 0x78)
    disable 1
    continue
end

run

Commands:

npm install @tensorflow/tfjs-node@4.22.0
node reproduce.js
gdb -q -batch -x force-counter-near-overflow.gdb --args node reproduce.js

Control result:

{"iteration":"complete","logicalTensorCount":0,"nativeTensorCount":0}

Counter-near-overflow result:

forced TFJSBackend::next_tensor_id_ to 2147483635
{"iteration":1,"logicalTensorCount":0,"nativeTensorCount":0}
{"iteration":2,"logicalTensorCount":0,"nativeTensorCount":1}
{"iteration":10,"logicalTensorCount":0,"nativeTensorCount":57}
{"iteration":100,"logicalTensorCount":0,"nativeTensorCount":687}
{"iteration":250,"logicalTensorCount":0,"nativeTensorCount":1737}
{"iteration":"complete","logicalTensorCount":0,"nativeTensorCount":1737}

After rollover, exactly seven native handles leak per loop while the logical count remains zero. The process exits normally.

Other information

Changing the JavaScript condition from id >= 0 to id !== -1 does not appear sufficient as a complete fix. It would allow deletion through the negative half-cycle, but continued rollover could eventually collide with IDs still used by persistent tensors.

A collision-safe reusable ID pool, or a wider end-to-end ID representation with an explicit non-ID sentinel, would avoid the cumulative allocation boundary. I can prepare a PR with the maintainers' preferred design and a regression test that forces the allocator near INT32_MAX, crosses the boundary, disposes the tensors, and verifies the native count returns to baseline.

[Varun-S10]

Hi @GideonKoenig, Thank you for raising this issue. We have reproduced the issue. As a workaround, add tf.backend().disposeData at application startup to allow calling binding.deleteTensor for negative tensor IDs following rollover. Could you please try this workaround and let us know if it resolves the issue for you?

[GideonKoenig]

Thanks for reproducing the issue. I tested the suggested workaround, and I want to confirm that I interpreted it correctly.

I interpreted it as overriding the active backend’s disposeData implementation at application startup, preserving the existing implementation but changing the native-deletion condition from id >= 0 to id !== -1:

- if (id != null && id >= 0) {
+ if (id != null && id !== -1) {
    this.binding.deleteTensor(id);
  }

Could you confirm that this is the intended workaround?

This resolves the immediate leak after the first signed rollover. Using the original reproducer:

Current implementation:
{"iteration":"complete","logicalTensorCount":0,"nativeTensorCount":1737}

With the disposeData override:
{"iteration":"complete","logicalTensorCount":0,"nativeTensorCount":0}

As a smaller side effect, the native allocator will eventually assign -1 as a real tensor ID. Because -1 is also the existing non-native sentinel, the workaround must skip it during disposal. This results in one leaked native handle when the counter passes through -1.

A single leaked handle per complete traversal of the ID space is unlikely to be significant by itself, but the sentinel conflict seems worth mentioning and suggests that -1 should be explicitly reserved by the native allocator.

There is a more serious problem once the counter finishes traversing the negative range and begins reusing positive IDs.

TFJSBackend::InsertHandle currently returns the iterator from unordered_map::insert without checking whether insertion succeeded:

return tfe_handle_map_
    .insert(std::make_pair(next_tensor_id_++, tfe_handle))
    .first->first;

If the candidate ID is still owned by a persistent tensor, insert does not overwrite that entry. Instead:

1. The existing native handle remains registered under that ID.
2. The newly created native handle is not inserted and becomes unreachable.
3. The existing ID is returned to JavaScript for the new tensor.
4. The persistent and temporary JavaScript tensors now both reference the same native ID.
5. Disposing the temporary tensor deletes the persistent tensor’s native handle.
6. The persistent JavaScript tensor remains alive but points to a missing native handle.

I reproduced this with a persistent tensor using native ID 0. After materializing that tensor, I forced the counter to -5. A temporary operation then allocates through -5, -4, -3, -2, -1, 0, causing the final allocation to collide with the persistent tensor.

The following reproduction assumes the disposeData override described above is installed.

Persistent-tensor collision reproducer

'use strict';

const tf = require('@tensorflow/tfjs-node');

function snapshot(label) {
  console.log(JSON.stringify({
    label,
    logicalTensorCount: tf.memory().numTensors,
    nativeTensorCount: tf.backend().getNumOfTFTensors(),
  }));
}

// Materializing this tensor assigns native ID 0. It represents a
// long-lived tensor such as a model weight.
const persistentWeight = tf.tensor1d([10, 20]);
const warmup = persistentWeight.square();

console.log(JSON.stringify({
  label: 'persistent-weight-created',
  weightValues: Array.from(persistentWeight.dataSync()),
  squareValues: Array.from(warmup.dataSync()),
}));

warmup.dispose();
snapshot('before-wrap');

try {
  tf.tidy(() => {
    const input = tf.tensor1d([1, 2, 3, 4]);
    const output = input.add(1).square().sub(input).sum();
    output.dataSync();
  });
} catch (error) {
  console.log(JSON.stringify({
    label: 'temporary-operation-error',
    message: error.message,
  }));
}

snapshot('after-temporary-disposal');

try {
  persistentWeight.dataSync();
} catch (error) {
  console.log(JSON.stringify({
    label: 'persistent-weight-error',
    message: error.message,
  }));
}

try {
  persistentWeight.dispose();
} catch (error) {
  console.log(JSON.stringify({
    label: 'persistent-weight-disposal-error',
    message: error.message,
  }));
}

snapshot('complete');

GDB script for the official 4.22.0 Linux x86-64 binary:

set pagination off
set confirm off
set breakpoint pending on
set debuginfod enabled off
set $insert_calls = 0

break tfnodejs::TFJSBackend::InsertHandle(TFE_TensorHandle*)
commands 1
    silent
    set $insert_calls = $insert_calls + 1
    if $insert_calls == 3
        # The persistent tensor owns ID 0. Start the temporary
        # allocation sequence at -5 so it reaches ID 0 again.
        set {int}($rdi + 0x78) = -5
        printf "forced next_tensor_id_ to %d with persistent ID 0 live\n", {int}($rdi + 0x78)
        disable 1
    end
    continue
end

run

Command:

gdb -q -batch \
  -x force-counter-through-collision.gdb \
  --args node reproduce-id-collision.js

Relevant output:

{"label":"before-wrap","logicalTensorCount":1,"nativeTensorCount":1}

forced next_tensor_id_ to -5 with persistent ID 0 live

{"label":"temporary-operation-error","message":"...Value for attr 'Tidx' of float is not in the list of allowed values..."}

{"label":"persistent-weight-error","message":"Get data called on a Tensor not referenced (tensor_id: 0)"}

{"label":"persistent-weight-disposal-error","message":"Delete called on a Tensor not referenced (tensor_id: 0)"}

{"label":"complete","logicalTensorCount":0,"nativeTensorCount":1}

The remaining reported native tensor is the handle assigned ID -1. The handle rejected by the colliding unordered_map::insert is not present in the map and therefore is not included in getNumOfTFTensors().

A more complete fix might therefore be:

1. Advance next_tensor_id_ with explicit wrapping, avoiding signed-overflow behavior.
2. Reserve and never allocate -1, since it is already used as the non-native sentinel.
3. Allow disposeData to delete all IDs other than -1.
4. Check the success result from tfe_handle_map_.insert.
5. If the candidate ID is already occupied, continue advancing until an unused ID is found.

The existing unordered_map::insert already performs the necessary lookup, so this should add very little overhead in the normal non-collision case.

If this approach matches the preferred design, I would be happy to prepare a PR containing the native allocator and corresponding disposal changes. If a reusable ID pool or wider ID representation would be preferred instead, I can work with that design as well.

[Varun-S10]

Hi @GideonKoenig, Thanks for the detailed analysis. Please go ahead and submit the Pull Request (PR). Once you open it, I will escalate it to the team for review.

[GideonKoenig]

Thanks — I've opened #8739 with the complete fix discussed above.

It explicitly handles the signed rollover, reserves -1 for non-native tensor data, disposes the remaining negative IDs, and retries allocation when a candidate ID is still owned by a live tensor.

I'm happy to adjust the implementation if the team would prefer a different allocator design.