mprof run occasionally fails with a TypeError

[pquentin]

mprof run occassionally fails with a TypeError. It happens, say one out of ten times. I'm on macOS and use memory_profiler 0.47 with psutil 5.3.1. Here's my traceback:

  File "venv/bin/mprof", line 554, in <module>
    actions[get_action()]()
  File "venv/bin/mprof", line 244, in run_action
    multiprocess=options.multiprocess, stream=f)
  File "venv/lib/python3.5/site-packages/memory_profiler.py", line 354, in memory_usage
    stream.write("MEM {0:.6f} {1:.4f}\n".format(*mem_usage))
TypeError: format() argument after * must be an iterable, not NoneType

I don't have a perfect reproducer but _get_memory can indeed return None with psutil on psutil.AccessDenied.

[stetelepta]

Same happens for me.. I'm on memory_profiler-0.52.0

[nwieder]

Same here, also on 0.52.0
edit1: seems to be somehow specific to functions that create child processes.
edit2: executing with the -m memory_profiler flag works.

[fabianp]

I gladly accept pull requests :-)

[astrojuanlu]

Could you please you provide a short, self-contained, correct code snippet that showcases the issue so we can try to reproduce it?

[mklosi]

still failing in 0.54.0

[astrojuanlu]

@mklosi can you please give us an example to reproduce this, as per my previous comment? I am sorry to say that "still failing" kind of messages are not helpful for us, so it's unlikely that we will be able to fix the issue without more information.

[pguridi]

When trying to run mprof run command, I'm getting the same error:
Traceback (most recent call last): File "/Users/pedro.guridi/.virtualenvs/processing_engine/bin/mprof", line 10, in <module> sys.exit(main()) File "/Users/pedro.guridi/.virtualenvs/processing_engine/lib/python3.7/site-packages/mprof.py", line 592, in main actions[get_action()]() File "/Users/pedro.guridi/.virtualenvs/processing_engine/lib/python3.7/site-packages/mprof.py", line 242, in run_action multiprocess=args.multiprocess, stream=f) File "/Users/pedro.guridi/.virtualenvs/processing_engine/lib/python3.7/site-packages/memory_profiler.py", line 363, in memory_usage stream.write("MEM {0:.6f} {1:.4f}\n".format(*mem_usage)) TypeError: format() argument after * must be an iterable, not NoneType

The command I'm using is: mprof run python my_script.py

• My script has a if __name__ == "__main__": entry point, and from there I'm executing my functions.
• Running my script without the profiler works perfectly.
• Its a single threaded and single process script.
• I'm using libraries such as Spacy (mentioning in case its relevant)
• This is running in a Python 3.7 virtualenvironment, on MacOS. And memory-profiler==0.55.0 installed using pip.

[pguridi]

Update here, I have a precise example to reproduce the crash:
running: mprof run test_fails.py:

file: test_fails.py

NLP = spacy.load("en_core_web_lg", disable=["parser", "tagger"])

if __name__ == "__main__":
    print("This doesn't work")

Output:

mprof run python test_fails.py
mprof: Sampling memory every 0.1s
This doesn't work
Traceback (most recent call last):
  File "/Users/pedro.guridi/.virtualenvs/processing_engine/bin/mprof", line 10, in <module>
    sys.exit(main())
  File "/Users/pedro.guridi/.virtualenvs/processing_engine/lib/python3.7/site-packages/mprof.py", line 592, in main
    actions[get_action()]()
  File "/Users/pedro.guridi/.virtualenvs/processing_engine/lib/python3.7/site-packages/mprof.py", line 242, in run_action
    multiprocess=args.multiprocess, stream=f)
  File "/Users/pedro.guridi/.virtualenvs/processing_engine/lib/python3.7/site-packages/memory_profiler.py", line 363, in memory_usage
    stream.write("MEM {0:.6f} {1:.4f}\n".format(*mem_usage))
TypeError: format() argument after * must be an iterable, not NoneType

To reproduce this error, install Spacy using pip, and add the data model with: python -m spacy download en_core_web_lg

[astrojuanlu]

Thanks @pguridi, that's useful. We'll try to give this a look soon.

[EmilRex]

Just wanted to chime in and say I ran across the same problem and subsequently found this discussion. I'm using version 0.55.0 on macOS (10.14.5). There may be other reasons as well, but one cause of this error stems from permissions. Running with sudo solved the issue for me. What happens is that the psutil.AccessDenied caught here causes _get_memory to return a None, which then cannot be unpacked in the stream.write call here. Not sure what solution you all would prefer, but two ideas are,

1. Return a list of dummy values (e.g. [0.0, 0.0]) instead of passing
2. Propagate the error and potentially also log a warning