TypeError: 'NoneType' object is not callable

[uduse]

🐛 Bug

(compiler_gym) zeyi@server121:~/scripts$ python main.py 
Exception in thread Thread-1:
Traceback (most recent call last):
  File ".../lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File ".../lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File ".../lib/python3.8/site-packages/aim/ext/resource/tracker.py", line 102, in _stat_collector
    self._track(stat)
  File ".../lib/python3.8/site-packages/aim/ext/resource/tracker.py", line 78, in _track
    self._track_func()(
TypeError: 'NoneType' object is not callable

To reproduce

Minimum example:

from aim import Run


def main():
    run = Run()
    for i in range(1000):
        run.track(i, name="loss")


if __name__ == "__main__":
    main()

This problem happens occasionally. In my case it happens one out of four times. However, if I change the script slightly, the problem above never appears.

from aim import Run

run = Run()

def main():
    for i in range(1000):
        run.track(i, name="loss")


if __name__ == "__main__":
    main()

Notice that I moved the variable run into global scope. I think the problem is due to some objects are de-referenced too quickly (e.g., the thread used to track? I'm not sure).

Expected behavior

The script should run without problem.

Environment

aim==3.4.1
aimrecords==0.0.7
aimrocks==0.0.10

Python 3.8.12

Linux server121 5.4.0-26-generic #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[alberttorosyan]

Hi @uduse!
I was not able to reproduce this behavior with aim v3.5.1. We made several bug fixes and improvements in this version and it may be fixed indirectly.
May I ask you to switch to the latest version and check if it solves the issue on your side?

[alberttorosyan]

Hey @uduse. Just wanted to check, did you have a chance to try (and possibly reproduce this issue) with aim v3.5.1+?
Thanks in advance.

[uduse]

@alberttorosyan Yeah I can't reproduce this problem with the new version of aim 👍 nice fix

[uduse]

I'm experiencing the same problem again. Though I can't reproduce the same problem in the small demo I shown above, this happens in my complicated project and I don't know what's going on...

Exception in thread Thread-34:
Traceback (most recent call last):
  File "/home/zeyi/anaconda3/envs/compiler_gym/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/zeyi/anaconda3/envs/compiler_gym/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/zeyi/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/aim/ext/resource/tracker.py", line 104, in _stat_collector
    self._track(stat)
  File "/home/zeyi/anaconda3/envs/compiler_gym/lib/python3.8/site-packages/aim/ext/resource/tracker.py", line 88, in _track
    self._track_func()(
TypeError: 'NoneType' object is not callable

[alberttorosyan]

@uduse let me share a bit of what's going on under the hood here. it might give you some ideas wrt how the more complex scenario might affect things.
so the resource tracking (when it's enabled) collects system resource usage info and tracks it as a regular metrics on the Run object. this is done in a separate thread, which is shut down when the Run is finished. it seems that the Run object is closed for writes before system tracking thread is shut down. is it possible that the Run.close/finalize methods are called explicitly?

[uduse]

No I don't call those functions explicitly. Could it be a concurrency problem? I'm having multiple async functions and each one of them has their own aim.Run object.

[sbuschjaeger]

I ran into this a few days ago with aim 3.19.3 and it was somewhat difficult to track down when using more than 1 thread for data loading.
Weirdly enough, it only occurred when using a smaller development data set (and not the entire dataset) and only after a sufficiently long runtime (e.g. when running 5 experiments, the first two finish and the third one randomly hangs).

Exception ignored in: <finalize object at 0x7f34d1c1ce60; dead>
Traceback (most recent call last):
  File "/usr/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File ".venv/sw/lib/python3.10/site-packages/aim/sdk/run.py", line 107, in _close
    res.close()
  File ".venv/sw/lib/python3.10/site-packages/aim/ext/resource/tracker.py", line 143, in close
    self.stop()
  File ".venv/sw/lib/python3.10/site-packages/aim/ext/resource/tracker.py", line 131, in stop
    self._th_collector.join()
  File "/usr/lib/python3.10/threading.py", line 1093, in join
    raise RuntimeError("cannot join current thread")
RuntimeError: cannot join current thread
2024-07-23 10:17:00,571 | INFO | BaseClassifier.py:182 | Star working on batch 0 for TEST
2024-07-23 10:17:00,576 | INFO | BaseClassifier.py:201 | Forward on batch 0 for TEST
Exception in thread Thread-721 (_stat_collector):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File ".venv/sw/lib/python3.10/site-packages/aim/ext/resource/tracker.py", line 189, in _stat_collector
    self._store_buffered_logs()
  File ".venv/sw/lib/python3.10/site-packages/aim/ext/resource/tracker.py", line 226, in _store_buffered_logs
    self._tracker()(LogLine(line.decode()), name='logs', step=self._line_counter)
TypeError: 'NoneType' object is not callable

I will update aim (although the changelog does not suggest any changes that might be relevant) and disable resource tracking.