Digits server won't start, device_query issue

[AliaMYH]

After working normally for a very long time, suddenly this error pops up when I try and run my digits server. I'm not sure why.
I ran device_query.py and posted the results as well.
@lukeyeager

$$$$$$$$$$:~/digits$ ./digits-devserver -p 5004
  ___ ___ ___ ___ _____ ___
 |   \_ _/ __|_ _|_   _/ __|
 | |) | | (_ || |  | | \__ \
 |___/___\___|___| |_| |___/ 5.1-dev

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/aliahassan95/digits/digits/__main__.py", line 70, in <module>
    main()
  File "/home/aliahassan95/digits/digits/__main__.py", line 55, in main
    import digits.webapp
  File "digits/webapp.py", line 64, in <module>
    import digits.model.images.classification.views  # noqa
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "digits/model/images/classification/views.py", line 12, in <module>
    from .forms import ImageClassificationModelForm
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "digits/model/images/classification/forms.py", line 4, in <module>
    from ..forms import ImageModelForm
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "digits/model/images/forms.py", line 6, in <module>
    from ..forms import ModelForm
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "digits/model/forms.py", line 18, in <module>
    class ModelForm(Form):
  File "digits/model/forms.py", line 321, in ModelForm
    ) for index in config_value('gpu_list').split(',') if index],
  File "digits/device_query.py", line 259, in get_nvml_info
    raise RuntimeError('nvmlDeviceGetHandleByPciBusId() failed with error #%s' % rc)
RuntimeError: nvmlDeviceGetHandleByPciBusId() failed with error #2


$$$$$$$:~/digits/digits$ ./device_query.py 
Device #0:
>>> CUDA attributes:
  name                         GeForce GTX TITAN X
  totalGlobalMem               12799574016
  clockRate                    1076000
  major                        5
  minor                        2
Traceback (most recent call last):
  File "./device_query.py", line 318, in <module>
    info = get_nvml_info(i)
  File "./device_query.py", line 259, in get_nvml_info
    raise RuntimeError('nvmlDeviceGetHandleByPciBusId() failed with error #%s' % rc)
RuntimeError: nvmlDeviceGetHandleByPciBusId() failed with error #2

[lukeyeager]

cudaErrorMemoryAllocation = 2
The API call failed because it was unable to allocate enough memory to perform the requested operation.
http://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038

Is there another process using up all the memory on your GPU?

[sumsuddin]

Thanks @lukeyeager it was a memory issue for me.