Problem with unicode characters in Slurm job names

[laparn]

Hello to all,

When jobs are submitted with names including unicode characters (perfectly allowed by slurm), then when doing a job.get on them, I get 👍

JobId=90 Name=dédé job
UserId=admin(9999) GroupId=admin(9999)
Priority=4294901746 Account=(null) QOS=(null)
JobState=COMPLETED Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
RunTime=00:00:01 TimeLimit=12:00:00 TimeMin=N/A
SubmitTime=2013-05-06T15:53:38 EligibleTime=2013-05-06T15:53:38
StartTime=2013-05-06T15:53:38 EndTime=2013-05-06T15:53:39
PreemptTime=None SuspendTime=None SecsPreSuspend=0
Partition=computes AllocNode:Sid=vpdev-master:24449
ReqNodeList=(null) ExcNodeList=(null)
NodeList=workstation-1-1
BatchHost=workstation-1-1
NumNodes=1 NumCPUs=1 CPUs/Task=1 ReqS:C:T=::*
MinCPUsNode=1 MinMemoryCPU=1024M MinTmpDiskNode=0
Features=(null) Gres=(null) Reservation=(null)
Shared=0 Contiguous=0 Licenses=(null) Network=(null)
Command=/tmp/tmppKlgGg
WorkDir=/tmp

import pyslurm
jobs=pyslurm.job()
jobs.get()
Traceback (most recent call last):
File "", line 1, in
File "pyslurm.pyx", line 1588, in pyslurm.pyslurm.job.get (pyslurm/pyslurm.c:20266)
File "pyslurm.pyx", line 1705, in pyslurm.pyslurm.job.__get (pyslurm/pyslurm.c:21489)
File "slurm.pxd", line 56, in pyslurm.slurm.stringOrNone (pyslurm/pyslurm.c:46456)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 13: ordinal not in range(128)

Clearly, this is something that pyslurm does not like much. I have not tested, but I suppose that unicode character in the workdir will have the same consequences.

[laparn]

Maybe it is not so complex. When looking at the corresponding code, what I see is :

cdef inline stringOrNone(char* value, value2):
    if value is NULL:
        if value2 is '':
            return None
        return u"%s" % value2
    return u"%s" % value

Line 56 is the last one.
So what is happening, is that when reading the characters (char *value), there is a character which is between 128 and 255, as the encoding is not known (ascii is the default in python), python does not know what to do, and raise an error. So, it must be found how to tell python what to do with these values or what is the encoding.

[laparn]

Finally, I found 2 solutions :

cdef inline stringOrNone(char* value, value2):
    if value is NULL:
        if value2 is '':
            return None
        return u"%s" % value2
    #return u"%s" % value.decode('Latin-1')
    return value.decode('ISO-8859-15')

When using you get :
arnaud@D3550:/src/slurm-2.5.0$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
736 debug dédé l arnaud PD 0:00 1 (Resources)
arnaud@D3550:/src/slurm-2.5.0$ python
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pyslurm
jobs=pyslurm.job()
a=jobs.get()
print a
{736: {u'comment': None, u'time_limit': 60L, u'cnode_cnt': None, u'alloc_node': u'D3550', u'features': [], u'eligible_time': 1367923231, u'contiguous': False, u'resv_id': None, u'ramdisk_image': None, u'block_id': None, u'sockets_per_node': 65534, u'req_switch': 0L, u'resv_name': None, u'licenses': {}, u'qos': None, u'submit_time': 1367923299, u'mloader_image': None, u'num_cpus': 1L, u'conn_type': (None, 'None'), u'show_flags': 0, u'user_id': 1001L, u'network': None, u'restart_cnt': 1, u'work_dir': u'/home/arnaud/src/slurmjob', u'pn_min_tmp_disk': 0L, u'max_nodes': 0L, u'job_state': (0, 'PENDING'), u'assoc_id': 0L, u'exit_code': 0L, u'num_nodes': 1L, u'priority': 4294901752L, u'batch_script': None, u'boards_per_node': 0, u'ntasks_per_socket': 65535, u'batch_flag': 1, u'derived_ec': 0L, u'nodes': None, u'preempt_time': 0, u'pn_min_cpus': 1, u'nice': 10000, u'ntasks_per_node': 0, u'linux_image': None, u'altered': None, u'sockets_per_board': 0, u'alloc_sid': 9968L, u'start_time': 1368011773, u'pre_sus_time': 0, u'ionodes': None, u'state_reason': (3, 'Resources'), u'pn_min_memory': 2147484648L, u'rotate': False, u'reboot': None, u'blrts_image': None, u'shared': 0, u'time_min': 0L, u'wait4switch': 0L, u'ntasks_per_core': 65535, u'wckey': None, u'account': None, u'requeue': True, u'name': u'd\xc3\xa9d\xc3\xa9 la sardine', u'req_nodes': [], u'gres': [], u'suspend_time': 0, u'partition': 'debug', u'cores_per_socket': 65534, u'batch_host': u'D3550', u'dependency': None, u'max_cpus': 0L, u'state_desc': None, u'command': u'/home/arnaud/src/slurmjob/./wait-arg.sh 60', u'end_time': 1367923299, u'cpus_per_task': 1, u'resize_time': 0, u'group_id': 1001L, u'exc_nodes': [], u'threads_per_core': 65534}}
print a[736]
{u'comment': None, u'time_limit': 60L, u'cnode_cnt': None, u'alloc_node': u'D3550', u'features': [], u'eligible_time': 1367923231, u'contiguous': False, u'resv_id': None, u'ramdisk_image': None, u'block_id': None, u'sockets_per_node': 65534, u'req_switch': 0L, u'resv_name': None, u'licenses': {}, u'qos': None, u'submit_time': 1367923299, u'mloader_image': None, u'num_cpus': 1L, u'conn_type': (None, 'None'), u'show_flags': 0, u'user_id': 1001L, u'network': None, u'restart_cnt': 1, u'work_dir': u'/home/arnaud/src/slurmjob', u'pn_min_tmp_disk': 0L, u'max_nodes': 0L, u'job_state': (0, 'PENDING'), u'assoc_id': 0L, u'exit_code': 0L, u'num_nodes': 1L, u'priority': 4294901752L, u'batch_script': None, u'boards_per_node': 0, u'ntasks_per_socket': 65535, u'batch_flag': 1, u'derived_ec': 0L, u'nodes': None, u'preempt_time': 0, u'pn_min_cpus': 1, u'nice': 10000, u'ntasks_per_node': 0, u'linux_image': None, u'altered': None, u'sockets_per_board': 0, u'alloc_sid': 9968L, u'start_time': 1368011773, u'pre_sus_time': 0, u'ionodes': None, u'state_reason': (3, 'Resources'), u'pn_min_memory': 2147484648L, u'rotate': False, u'reboot': None, u'blrts_image': None, u'shared': 0, u'time_min': 0L, u'wait4switch': 0L, u'ntasks_per_core': 65535, u'wckey': None, u'account': None, u'requeue': True, u'name': u'd\xc3\xa9d\xc3\xa9 la sardine', u'req_nodes': [], u'gres': [], u'suspend_time': 0, u'partition': 'debug', u'cores_per_socket': 65534, u'batch_host': u'D3550', u'dependency': None, u'max_cpus': 0L, u'state_desc': None, u'command': u'/home/arnaud/src/slurmjob/./wait-arg.sh 60', u'end_time': 1367923299, u'cpus_per_task': 1, u'resize_time': 0, u'group_id': 1001L, u'exc_nodes': [], u'threads_per_core': 65534}
print a[736]['name']
dédé la sardine
print a[736]['name'].encode('Latin-1')
dédé la sardine
type(a[736]['name'])
<type 'unicode'>
b= a[736]['name'].encode('Latin-1')
print b
dédé la sardine
b
'd\xc3\xa9d\xc3\xa9 la sardine'

So finally, the 'name' entry for a job is real unicode. However to display it, you need to "announce" the kind of encoding that is used, hence the encode('Latin-1').

The second solution is to switch directly to a str :

cdef inline stringOrNone(char* value, value2):
    if value is NULL:
        if value2 is '':
            return None
        return u"%s" % value2
    #return u"%s" % value.decode('Latin-1')
    return value.decode('ISO-8859-15').encode('ISO-8859-15')

Then when using, you have :
arnaud@D3550:/src/slurmjob$ make unicode
sbatch -J "dédé la sardine" -N 1 ./wait-arg.sh 60
Submitted batch job 737
arnaud@D3550:/src/slurmjob$ python
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pyslurm
import pyslurm
jobs=pyslurm.job()
jobs.get()
{736: {u'comment': None, u'time_limit': 60L, u'cnode_cnt': None, u'alloc_node': 'D3550', u'features': [], u'eligible_time': 1367923231, u'contiguous': False, u'resv_id': None, u'ramdisk_image': None, u'block_id': None, u'sockets_per_node': 65534, u'req_switch': 0L, u'resv_name': None, u'licenses': {}, u'qos': None, u'submit_time': 1367923299, u'mloader_image': None, u'num_cpus': 1L, u'conn_type': (None, 'None'), u'show_flags': 0, u'user_id': 1001L, u'network': None, u'restart_cnt': 1, u'work_dir': '/home/arnaud/src/slurmjob', u'pn_min_tmp_disk': 0L, u'max_nodes': 0L, u'job_state': (0, 'PENDING'), u'assoc_id': 0L, u'exit_code': 0L, u'num_nodes': 1L, u'priority': 4294901752L, u'batch_script': None, u'boards_per_node': 0, u'ntasks_per_socket': 65535, u'batch_flag': 1, u'derived_ec': 0L, u'nodes': None, u'preempt_time': 0, u'pn_min_cpus': 1, u'nice': 10000, u'ntasks_per_node': 0, u'linux_image': None, u'altered': None, u'sockets_per_board': 0, u'alloc_sid': 9968L, u'start_time': 1368015717, u'pre_sus_time': 0, u'ionodes': None, u'state_reason': (3, 'Resources'), u'pn_min_memory': 2147484648L, u'rotate': False, u'reboot': None, u'blrts_image': None, u'shared': 0, u'time_min': 0L, u'wait4switch': 0L, u'ntasks_per_core': 65535, u'wckey': None, u'account': None, u'requeue': True, u'name': 'd\xc3\xa9d\xc3\xa9 la sardine', u'req_nodes': [], u'gres': [], u'suspend_time': 0, u'partition': 'debug', u'cores_per_socket': 65534, u'batch_host': 'D3550', u'dependency': None, u'max_cpus': 0L, u'state_desc': None, u'command': '/home/arnaud/src/slurmjob/./wait-arg.sh 60', u'end_time': 1367923299, u'cpus_per_task': 1, u'resize_time': 0, u'group_id': 1001L, u'exc_nodes': [], u'threads_per_core': 65534}, 737: {u'comment': None, u'time_limit': 60L, u'cnode_cnt': None, u'alloc_node': 'D3550', u'features': [], u'eligible_time': 1367929295, u'contiguous': False, u'resv_id': None, u'ramdisk_image': None, u'block_id': None, u'sockets_per_node': 65534, u'req_switch': 0L, u'resv_name': None, u'licenses': {}, u'qos': None, u'submit_time': 1367929295, u'mloader_image': None, u'num_cpus': 1L, u'conn_type': (None, 'None'), u'show_flags': 0, u'user_id': 1001L, u'network': None, u'restart_cnt': 0, u'work_dir': '/home/arnaud/src/slurmjob', u'pn_min_tmp_disk': 0L, u'max_nodes': 1L, u'job_state': (0, 'PENDING'), u'assoc_id': 0L, u'exit_code': 0L, u'num_nodes': 1L, u'priority': 4294901751L, u'batch_script': None, u'boards_per_node': 0, u'ntasks_per_socket': 65535, u'batch_flag': 1, u'derived_ec': 0L, u'nodes': None, u'preempt_time': 0, u'pn_min_cpus': 1, u'nice': 10000, u'ntasks_per_node': 0, u'linux_image': None, u'altered': None, u'sockets_per_board': 0, u'alloc_sid': 9426L, u'start_time': 1368015717, u'pre_sus_time': 0, u'ionodes': None, u'state_reason': (3, 'Resources'), u'pn_min_memory': 2147484648L, u'rotate': False, u'reboot': None, u'blrts_image': None, u'shared': 0, u'time_min': 0L, u'wait4switch': 0L, u'ntasks_per_core': 65535, u'wckey': None, u'account': None, u'requeue': True, u'name': 'd\xc3\xa9d\xc3\xa9 la sardine', u'req_nodes': [], u'gres': [], u'suspend_time': 0, u'partition': 'debug', u'cores_per_socket': 65534, u'batch_host': None, u'dependency': None, u'max_cpus': 0L, u'state_desc': None, u'command': '/home/arnaud/src/slurmjob/./wait-arg.sh 60', u'end_time': 0, u'cpus_per_task': 1, u'resize_time': 0, u'group_id': 1001L, u'exc_nodes': [], u'threads_per_core': 65534}}

a=jobs.get()
print a[736]['name']
dédé la sardine
a[736]['name']
'd\xc3\xa9d\xc3\xa9 la sardine'
type(a[736]['name'] )
<type 'str'>

Well, from my point of view, it is more "natural" not to have to use the encode for the display. The real problem that I see is that I do not know what is the real encoding used within slurm. It is definitly unclear. The messages are encoded with one byte per character, using both Latin-1 or ISO-8859-15 are working on my machine (ubuntu 12.04, python 2.7, slurm 2.5.0). But Ubuntu is in full utf8 (import sys;sys.getfilesystemencoding();).

The best solution would be to use the encoding provided by sys.getfilesystemencoding() for the encode/decode operation. However, when doing that in the pxd, it introduces an error at the load of pyslurm. So I try finally :

cdef inline stringOrNone(char* value, value2):
    if value is NULL:
        if value2 is '':
            return None
        return u"%s" % value2
    #return u"%s" % value.decode('Latin-1')
    #current_encoding=sys.getfilesystemencoding()
    return value.decode('UTF-8')

And when using it :

import pyslurm
jobs=pyslurm.job()
jobs.get()
{736: {u'comment': None, u'time_limit': 60L, u'cnode_cnt': None, u'alloc_node': u'D3550', u'features': [], u'eligible_time': 1367923231, u'contiguous': False, u'resv_id': None, u'ramdisk_image': None, u'block_id': None, u'sockets_per_node': 65534, u'req_switch': 0L, u'resv_name': None, u'licenses': {}, u'qos': None, u'submit_time': 1367923299, u'mloader_image': None, u'num_cpus': 1L, u'conn_type': (None, 'None'), u'show_flags': 0, u'user_id': 1001L, u'network': None, u'restart_cnt': 1, u'work_dir': u'/home/arnaud/src/slurmjob', u'pn_min_tmp_disk': 0L, u'max_nodes': 0L, u'job_state': (0, 'PENDING'), u'assoc_id': 0L, u'exit_code': 0L, u'num_nodes': 1L, u'priority': 4294901752L, u'batch_script': None, u'boards_per_node': 0, u'ntasks_per_socket': 65535, u'batch_flag': 1, u'derived_ec': 0L, u'nodes': None, u'preempt_time': 0, u'pn_min_cpus': 1, u'nice': 10000, u'ntasks_per_node': 0, u'linux_image': None, u'altered': None, u'sockets_per_board': 0, u'alloc_sid': 9968L, u'start_time': 1368017921, u'pre_sus_time': 0, u'ionodes': None, u'state_reason': (3, 'Resources'), u'pn_min_memory': 2147484648L, u'rotate': False, u'reboot': None, u'blrts_image': None, u'shared': 0, u'time_min': 0L, u'wait4switch': 0L, u'ntasks_per_core': 65535, u'wckey': None, u'account': None, u'requeue': True, u'name': u'd\xe9d\xe9 la sardine', u'req_nodes': [], u'gres': [], u'suspend_time': 0, u'partition': 'debug', u'cores_per_socket': 65534, u'batch_host': u'D3550', u'dependency': None, u'max_cpus': 0L, u'state_desc': None, u'command': u'/home/arnaud/src/slurmjob/./wait-arg.sh 60', u'end_time': 1367923299, u'cpus_per_task': 1, u'resize_time': 0, u'group_id': 1001L, u'exc_nodes': [], u'threads_per_core': 65534}, 737: {u'comment': None, u'time_limit': 60L, u'cnode_cnt': None, u'alloc_node': u'D3550', u'features': [], u'eligible_time': 1367929295, u'contiguous': False, u'resv_id': None, u'ramdisk_image': None, u'block_id': None, u'sockets_per_node': 65534, u'req_switch': 0L, u'resv_name': None, u'licenses': {}, u'qos': None, u'submit_time': 1367929295, u'mloader_image': None, u'num_cpus': 1L, u'conn_type': (None, 'None'), u'show_flags': 0, u'user_id': 1001L, u'network': None, u'restart_cnt': 0, u'work_dir': u'/home/arnaud/src/slurmjob', u'pn_min_tmp_disk': 0L, u'max_nodes': 1L, u'job_state': (0, 'PENDING'), u'assoc_id': 0L, u'exit_code': 0L, u'num_nodes': 1L, u'priority': 4294901751L, u'batch_script': None, u'boards_per_node': 0, u'ntasks_per_socket': 65535, u'batch_flag': 1, u'derived_ec': 0L, u'nodes': None, u'preempt_time': 0, u'pn_min_cpus': 1, u'nice': 10000, u'ntasks_per_node': 0, u'linux_image': None, u'altered': None, u'sockets_per_board': 0, u'alloc_sid': 9426L, u'start_time': 1368017921, u'pre_sus_time': 0, u'ionodes': None, u'state_reason': (3, 'Resources'), u'pn_min_memory': 2147484648L, u'rotate': False, u'reboot': None, u'blrts_image': None, u'shared': 0, u'time_min': 0L, u'wait4switch': 0L, u'ntasks_per_core': 65535, u'wckey': None, u'account': None, u'requeue': True, u'name': u'd\xe9d\xe9 la sardine', u'req_nodes': [], u'gres': [], u'suspend_time': 0, u'partition': 'debug', u'cores_per_socket': 65534, u'batch_host': None, u'dependency': None, u'max_cpus': 0L, u'state_desc': None, u'command': u'/home/arnaud/src/slurmjob/./wait-arg.sh 60', u'end_time': 0, u'cpus_per_task': 1, u'resize_time': 0, u'group_id': 1001L, u'exc_nodes': [], u'threads_per_core': 65534}, 738: {u'comment': None, u'time_limit': 60L, u'cnode_cnt': None, u'alloc_node': u'D3550', u'features': [], u'eligible_time': 1367931528, u'contiguous': False, u'resv_id': None, u'ramdisk_image': None, u'block_id': None, u'sockets_per_node': 65534, u'req_switch': 0L, u'resv_name': None, u'licenses': {}, u'qos': None, u'submit_time': 1367931528, u'mloader_image': None, u'num_cpus': 1L, u'conn_type': (None, 'None'), u'show_flags': 0, u'user_id': 1001L, u'network': None, u'restart_cnt': 0, u'work_dir': u'/home/arnaud/src/slurmjob', u'pn_min_tmp_disk': 0L, u'max_nodes': 1L, u'job_state': (0, 'PENDING'), u'assoc_id': 0L, u'exit_code': 0L, u'num_nodes': 1L, u'priority': 4294901750L, u'batch_script': None, u'boards_per_node': 0, u'ntasks_per_socket': 65535, u'batch_flag': 1, u'derived_ec': 0L, u'nodes': None, u'preempt_time': 0, u'pn_min_cpus': 1, u'nice': 10000, u'ntasks_per_node': 0, u'linux_image': None, u'altered': None, u'sockets_per_board': 0, u'alloc_sid': 9426L, u'start_time': 0, u'pre_sus_time': 0, u'ionodes': None, u'state_reason': (3, 'Resources'), u'pn_min_memory': 2147484648L, u'rotate': False, u'reboot': None, u'blrts_image': None, u'shared': 0, u'time_min': 0L, u'wait4switch': 0L, u'ntasks_per_core': 65535, u'wckey': None, u'account': None, u'requeue': True, u'name': u'd\xe9d\xe9 la sardine', u'req_nodes': [], u'gres': [], u'suspend_time': 0, u'partition': 'debug', u'cores_per_socket': 65534, u'batch_host': None, u'dependency': None, u'max_cpus': 0L, u'state_desc': None, u'command': u'/home/arnaud/src/slurmjob/./wait-arg.sh 60', u'end_time': 0, u'cpus_per_task': 1, u'resize_time': 0, u'group_id': 1001L, u'exc_nodes': [], u'threads_per_core': 65534}}
a[736]['name']
u'd\xe9d\xe9 la sardine'
print a[736]['name']
dédé la sardine
type(a[736]['name'])
<type 'unicode'>

This looks good to me. I think (from the code) that what comes out of a[736]['name'] should be unicode, and at the same time, it prints nicely. I still would prefer to take the encoding information of sys.getfilesystemencoding(). If somebody knows how to do ...

This is just my proposition. Could somebody test it again in another environment and push it to git if it is ok ?

[gingergeeks]

Agreed :) I made that display choice for a quick fix for something I can't quite remember now ! I will make the suggested changes once I get the new code merges of the new test framework from supermanue.

[gingergeeks]

Closing this for now as we plan Python3 and unicode shortly. Remi (EDF) has recently supplied a quick patch so we plan to merge that as well.