UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 31798: invalid continuation byte

[mgdondon]

Hello,

It seems to be a bug in version 2.2.7

• OS: Windows 7 - 32 bits
• SAS Version: 9.4M1
• SASPy Version: 2.2.7

Import in Python

import pandas as pd
data_url = "https://forge.scilab.org/index.php/p/rdataset/source/file/master/csv/datasets/cars.csv"
df_python = pd.read_csv(data_url)
df_python.drop(df_python.columns[[0]], axis=1, inplace=True)

Summary in SAS

import saspy
session_sas = saspy.SASsession(cfgname='winlocal')

SAS Connection established. Subprocess id is 6920

session_sas

Access Method         = IOM
SAS Config name       = winlocal
WORK Path             = /
SAS Version           = 9.04.01M1P12042013
SASPy Version         = 2.2.7
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = WLATIN1
Python Encoding value = windows-1252

df_sas = session_sas.df2sd(df_python)
df_sas.describe()

---------------------------------------------------------------------------

UnicodeDecodeError                        Traceback (most recent call last)

<ipython-input-4-efc5eac237bd> in <module>()
      1 df_sas = session_sas.df2sd(df_python)
----> 2 df_sas.describe()

f:\program files\python\python37-32\lib\site-packages\saspy\sasbase.py in describe(self)
   1508         :return:
   1509         """
-> 1510         return self.means()
   1511 
   1512     def means(self):

f:\program files\python\python37-32\lib\site-packages\saspy\sasbase.py in means(self)
   1527             code = "proc means data=%s.%s %s stackodsoutput n nmiss median mean std min p25 p50 p75 max; ods output Summary=work._summary; run;" % (
   1528                 self.libref, self.table, self._dsopts())
-> 1529             return self._returnPD(code, '_summary')
   1530         else:
   1531             if self.HTML:

f:\program files\python\python37-32\lib\site-packages\saspy\sasbase.py in _returnPD(self, code, tablename, **kwargs)
   1107         if 'libref' in kwargs:
   1108             libref = kwargs['libref']
-> 1109         ll = self.sas._io.submit(code)
   1110         check, errorMsg = self._checkLogForError(ll['LOG'])
   1111         if not check:

f:\program files\python\python37-32\lib\site-packages\saspy\sasioiom.py in submit(self, code, results, prompt)
    875              self.stdin[0].send(odsclose+logcodei.encode()+b'tom says EOL='+logcodeo+b'\n')
    876 
--> 877       lstf = lstf.decode()
    878       logf = logf.decode()
    879 

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 31798: invalid continuation byte

---

which is not in version 2.2.6

• OS: Windows 7 - 32 bits
• SAS Version: 9.4M1
• SASPy Version: 2.2.6

import saspy
session_sas = saspy.SASsession(cfgname='winlocal')

SAS Connection established. Subprocess id is 6784

session_sas

Access Method         = IOM
SAS Config name       = winlocal
WORK Path             = /
SAS Version           = 9.04.01M1P12042013
SASPy Version         = 2.2.6
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = WLATIN1
Python Encoding value = windows-1252

df_sas = session_sas.df2sd(df_python)
df_sas.describe()

	Variable	N	NMiss	Median	Mean	StdDev	Min	P25	P50	P75	Max
0	speed	50	0	15	15.40	5.287644	4	12	15	19	25
1	dist	50	0	36	42.98	25.769377	2	26	36	56	120

---

Two more remarks:

• I get only log outputs when I use SASKernel with SASPy bug. SASKernel works fine when SASPy is OK.
• The following configuration is also OK with SASPy and SASKernel
  • OS: Windows 7 - 64 bits
  • SAS Version: 9.4M3
  • SASPy Version: 2.2.7

import saspy
session_sas = saspy.SASsession(cfgname='winlocal')

SAS Connection established. Subprocess id is 5676

session_sas

Access Method         = IOM
SAS Config name       = winlocal
WORK Path             = /
SAS Version           = 9.04.01M3P06242015
SASPy Version         = 2.2.7
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = WLATIN1
Python Encoding value = windows-1252

df_sas = session_sas.df2sd(df_python)
df_sas.describe()

	Variable	N	NMiss	Median	Mean	StdDev	Min	P25	P50	P75	Max
0	speed	50	0	15	15.40	5.287644	4	12	15	19	25
1	dist	50	0	36	42.98	25.769377	2	26	36	56	120

Thanks,

[tomweber-sas]

@mgdondon great bug writeup! I am able to run this and I've posted my results to the new saspy-examples/Issues folder
https://github.com/sassoftware/saspy-examples/

I'm running 9.4m5, and am, of course, on a 64 bit OS. I should be able to try out 9.4m1, but I don't think I'll be able to run anything on 32bits. I'm a little suspicions of that. I'm curious why the workpath didn't get resolved right, but I should see that in the log.

There are 2 things I'd like you to do, if you would. The first is to

print(session_sas.saslog())

so we can see if there was any kind of error or anything strange on the SAS side for this. I have that in my run (https://github.com/sassoftware/saspy-examples/blob/master/Issue_examples/Issue168.ipynb).

The other thing is to try the sasioiom.py that's at master. I had some strange errors like this with translated version of SAS in Chinese which I also can't reproduce here, that I've been trying to track down. That file at master had a little different path for that lstf.decode(), so I think that will get a result, which may show us what we're getting in that lst that is throwing the error.

Can you run this with the saslog output, with your existing 2.2.7, then again w/ master?

Thanks!
Tom

[tomweber-sas]

One other thought. The saspyiom.jar file changed in 2.2.7, from 2.2.6. Can you make sure that the classpath when running 2.2.7 is pointing to the new jar? Just to be sure :)
Tom

[mgdondon]

SAS log with SASPy 2.2.7

Import in Python

import pandas as pd
data_url = "https://forge.scilab.org/index.php/p/rdataset/source/file/master/csv/datasets/cars.csv"
df_python = pd.read_csv(data_url)
df_python.drop(df_python.columns[[0]], axis=1, inplace=True)

Summary in SAS

import saspy
session_sas = saspy.SASsession(cfgname='winlocal')

SAS Connection established. Subprocess id is 5400

session_sas

Access Method         = IOM
SAS Config name       = winlocal
WORK Path             = /
SAS Version           = 9.04.01M1P12042013
SASPy Version         = 2.2.7
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = WLATIN1
Python Encoding value = windows-1252

df_sas = session_sas.df2sd(df_python)

print(session_sas.saslog())

1                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA. 
NOTE: SAS (r) Proprietary Software 9.4 (TS1M1) 
      Licensed to LICENCE MESR SAS RECHERCHE PCW, Site 70109801.
NOTE: This session is executing on the W32_7PRO  platform.
  
NOTE: Updated analytical products:
      
      SAS/STAT 13.1
      SAS/ETS 13.1
      SAS/OR 13.1
      SAS/IML 13.1
      SAS/QC 13.1

NOTE: Additional host information:

 W32_7PRO WIN 6.1.7601 Service Pack 1 Workstation

NOTE: Initialisation de SAS used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
1          ;*';*";*/;
2          options svgtitle='svgtitle'; options validvarname=any pagesize=max nosyntaxcheck; ods graphics on;
3          
4          ;*';*";*/;
5          %put E3969440A681A2408885998500000001;
E3969440A681A2408885998500000001
6          
2                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

7          ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
7        ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
8          ;*';*";*/;
9          libname work list;
NOTE: Libref=   WORK 
      Scope=    IOM ROOT COMP ENV
      Moteur=   V9
      Access=   TEMP
      Nom physique= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD4212_PMGD_\Prc2
      Nom du fichier= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD4212_PMGD_\Prc2
10         
11         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
12         
13         %put E3969440A681A2408885998500000002;
E3969440A681A2408885998500000002
14         
3                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

15         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
15       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
16         ;*';*";*/;
17         %put SYSV=&sysvlong4;
SYSV=9.04.01M1P12042013
18         
19         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
20         
21         %put E3969440A681A2408885998500000003;
E3969440A681A2408885998500000003
22         
4                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

23         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
23       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
24         ;*';*";*/;
25         proc options option=encoding;run;

    SAS (r) Proprietary Software Version 9.4  TS1M1

 ENCODING=WLATIN1  Specifies the default character-set encoding for the SAS session.
NOTE: PROCEDURE OPTIONS used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
26         
27         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
28         
29         %put E3969440A681A2408885998500000004;
E3969440A681A2408885998500000004
30         
5                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

31         data _df;
32         length  'speed'n 8 'dist'n 8;
33         infile datalines delimiter='03'x DSD STOPOVER;
34         input @;
35         if _infile_ = '' then delete;
36         input 'speed'n 'dist'n ;
37         datalines4;

6                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

NOTE: DATA statement used (Total process time):
      real time           0.06 secondes
      cpu time            0.00 secondes
      
NOTE: The data set WORK._DF has 50 observations and 2 variables.

89         ;;;;
90         run;
91         
7                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

92         ;*';*";*/;
93         
94         
95         ;*';*";*/;
96         %put E3969440A681A2408885998500000005;
E3969440A681A2408885998500000005
97         
8                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

98         ;*';*";*/;
99         data _null_; e = exist('_df');
100        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 1
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
101        
102        
103        ;*';*";*/;
104        %put E3969440A681A2408885998500000006;
E3969440A681A2408885998500000006
105        
9                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

106        ;*';*";*/;
107        data _null_; e = exist('user._df');
108        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 0
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
109        
110        
111        ;*';*";*/;
112        %put E3969440A681A2408885998500000007;
E3969440A681A2408885998500000007
113        

df_sas.describe()

---------------------------------------------------------------------------

UnicodeDecodeError                        Traceback (most recent call last)

<ipython-input-6-503b40ae1143> in <module>()
----> 1 df_sas.describe()

f:\program files\python\python37-32\lib\site-packages\saspy\sasbase.py in describe(self)
   1508         :return:
   1509         """
-> 1510         return self.means()
   1511 
   1512     def means(self):

f:\program files\python\python37-32\lib\site-packages\saspy\sasbase.py in means(self)
   1527             code = "proc means data=%s.%s %s stackodsoutput n nmiss median mean std min p25 p50 p75 max; ods output Summary=work._summary; run;" % (
   1528                 self.libref, self.table, self._dsopts())
-> 1529             return self._returnPD(code, '_summary')
   1530         else:
   1531             if self.HTML:

f:\program files\python\python37-32\lib\site-packages\saspy\sasbase.py in _returnPD(self, code, tablename, **kwargs)
   1107         if 'libref' in kwargs:
   1108             libref = kwargs['libref']
-> 1109         ll = self.sas._io.submit(code)
   1110         check, errorMsg = self._checkLogForError(ll['LOG'])
   1111         if not check:

f:\program files\python\python37-32\lib\site-packages\saspy\sasioiom.py in submit(self, code, results, prompt)
    875              self.stdin[0].send(odsclose+logcodei.encode()+b'tom says EOL='+logcodeo+b'\n')
    876 
--> 877       lstf = lstf.decode()
    878       logf = logf.decode()
    879 

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 31798: invalid continuation byte

print(session_sas.saslog())

1                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA. 
NOTE: SAS (r) Proprietary Software 9.4 (TS1M1) 
      Licensed to LICENCE MESR SAS RECHERCHE PCW, Site 70109801.
NOTE: This session is executing on the W32_7PRO  platform.

NOTE: Updated analytical products:
      
      SAS/STAT 13.1
      SAS/ETS 13.1
      SAS/OR 13.1
      SAS/IML 13.1
      SAS/QC 13.1

NOTE: Additional host information:

 W32_7PRO WIN 6.1.7601 Service Pack 1 Workstation

NOTE: Initialisation de SAS used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
1          ;*';*";*/;
2          options svgtitle='svgtitle'; options validvarname=any pagesize=max nosyntaxcheck; ods graphics on;
3          
4          ;*';*";*/;
5          %put E3969440A681A2408885998500000001;
E3969440A681A2408885998500000001
6          
2                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

7          ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
7        ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
8          ;*';*";*/;
9          libname work list;
NOTE: Libref=   WORK 
      Scope=    IOM ROOT COMP ENV
      Moteur=   V9
      Access=   TEMP
      Nom physique= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD4212_PMGD_\Prc2
      Nom du fichier= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD4212_PMGD_\Prc2
10         
11         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
12         
13         %put E3969440A681A2408885998500000002;
E3969440A681A2408885998500000002
14         
3                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

15         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
15       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
16         ;*';*";*/;
17         %put SYSV=&sysvlong4;
SYSV=9.04.01M1P12042013
18         
19         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
20         
21         %put E3969440A681A2408885998500000003;
E3969440A681A2408885998500000003
22         
4                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

23         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
23       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
24         ;*';*";*/;
25         proc options option=encoding;run;

    SAS (r) Proprietary Software Version 9.4  TS1M1

 ENCODING=WLATIN1  Specifies the default character-set encoding for the SAS session.
NOTE: PROCEDURE OPTIONS used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
26         
27         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
28         
29         %put E3969440A681A2408885998500000004;
E3969440A681A2408885998500000004
30         
5                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

31         data _df;
32         length  'speed'n 8 'dist'n 8;
33         infile datalines delimiter='03'x DSD STOPOVER;
34         input @;
35         if _infile_ = '' then delete;
36         input 'speed'n 'dist'n ;
37         datalines4;

6                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

NOTE: DATA statement used (Total process time):
      real time           0.06 secondes
      cpu time            0.00 secondes
      
NOTE: The data set WORK._DF has 50 observations and 2 variables.

89         ;;;;
90         run;
91         
7                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

92         ;*';*";*/;
93         
94         
95         ;*';*";*/;
96         %put E3969440A681A2408885998500000005;
E3969440A681A2408885998500000005
97         
8                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

98         ;*';*";*/;
99         data _null_; e = exist('_df');
100        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 1
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
101        
102        
103        ;*';*";*/;
104        %put E3969440A681A2408885998500000006;
E3969440A681A2408885998500000006
105        
9                                                          Le Système SAS                        16:21 Wednesday, September 12, 2018

106        ;*';*";*/;
107        data _null_; e = exist('user._df');
108        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 0
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
109        
110        
111        ;*';*";*/;
112        %put E3969440A681A2408885998500000007;
E3969440A681A2408885998500000007
113        
10                                                         Le Système SAS                        16:21 Wednesday, September 12, 2018

114        ;*';*";*/;
115        data _null_; e = exist('WORK._df');
116        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 1
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
117        
118        
119        ;*';*";*/;
120        %put E3969440A681A2408885998500000008;
E3969440A681A2408885998500000008
121        

[tomweber-sas]

Ah, this is also a translated version of SAS! That's why the WORK Path isn't getting found. And, no LOG for the proc means, as that would have been done after the error, so it's not in there.
I do suspect that you will get past this error and get the log for the proc means and data transfer with the code at master.

[mgdondon]

It's all right with sasioiom.py that's at master and SASKernel is also OK.

Thank you very much,

SAS log with SASPy 2.2.7 and sasioiom.py at master

Import in Python

import pandas as pd
data_url = "https://forge.scilab.org/index.php/p/rdataset/source/file/master/csv/datasets/cars.csv"
df_python = pd.read_csv(data_url)
df_python.drop(df_python.columns[[0]], axis=1, inplace=True)

Summary in SAS

import saspy
session_sas = saspy.SASsession(cfgname='winlocal')

SAS Connection established. Subprocess id is 4504

session_sas

Access Method         = IOM
SAS Config name       = winlocal
WORK Path             = /
SAS Version           = 9.04.01M1P12042013
SASPy Version         = 2.2.7
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = WLATIN1
Python Encoding value = windows-1252

df_sas = session_sas.df2sd(df_python)

print(session_sas.saslog())

1                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA. 
NOTE: SAS (r) Proprietary Software 9.4 (TS1M1) 
      Licensed to LICENCE MESR SAS RECHERCHE PCW, Site 70109801.
NOTE: This session is executing on the W32_7PRO  platform.

NOTE: Updated analytical products:
      
      SAS/STAT 13.1
      SAS/ETS 13.1
      SAS/OR 13.1
      SAS/IML 13.1
      SAS/QC 13.1

NOTE: Additional host information:

 W32_7PRO WIN 6.1.7601 Service Pack 1 Workstation

NOTE: Initialisation de SAS used (Total process time):
      real time           0.01 secondes
      cpu time            0.01 secondes
      
1          ;*';*";*/;
2          options svgtitle='svgtitle'; options validvarname=any pagesize=max nosyntaxcheck; ods graphics on;
3          
4          ;*';*";*/;
5          %put E3969440A681A2408885998500000001;
E3969440A681A2408885998500000001
6          
2                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

7          ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
7        ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
8          ;*';*";*/;
9          libname work list;
NOTE: Libref=   WORK 
      Scope=    IOM ROOT COMP ENV
      Moteur=   V9
      Access=   TEMP
      Nom physique= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD2492_PMGD_\Prc2
      Nom du fichier= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD2492_PMGD_\Prc2
10         
11         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
12         
13         %put E3969440A681A2408885998500000002;
E3969440A681A2408885998500000002
14         
3                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

15         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
15       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
16         ;*';*";*/;
17         %put SYSV=&sysvlong4;
SYSV=9.04.01M1P12042013
18         
19         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
20         
21         %put E3969440A681A2408885998500000003;
E3969440A681A2408885998500000003
22         
4                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

23         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
23       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
24         ;*';*";*/;
25         proc options option=encoding;run;

    SAS (r) Proprietary Software Version 9.4  TS1M1

 ENCODING=WLATIN1  Specifies the default character-set encoding for the SAS session.
NOTE: PROCEDURE OPTIONS used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
26         
27         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
28         
29         %put E3969440A681A2408885998500000004;
E3969440A681A2408885998500000004
30         
5                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

31         data _df;
32         length  'speed'n 8 'dist'n 8;
33         infile datalines delimiter='03'x DSD STOPOVER;
34         input @;
35         if _infile_ = '' then delete;
36         input 'speed'n 'dist'n ;
37         datalines4;

6                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

NOTE: DATA statement used (Total process time):
      real time           0.06 secondes
      cpu time            0.01 secondes
      
NOTE: The data set WORK._DF has 50 observations and 2 variables.

89         ;;;;
90         run;
91         
7                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

92         ;*';*";*/;
93         
94         
95         ;*';*";*/;
96         %put E3969440A681A2408885998500000005;
E3969440A681A2408885998500000005
97         
8                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

98         ;*';*";*/;
99         data _null_; e = exist('_df');
100        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 1
NOTE: DATA statement used (Total process time):
      real time           0.01 secondes
      cpu time            0.00 secondes
      

101        
102        
103        ;*';*";*/;
104        %put E3969440A681A2408885998500000006;
E3969440A681A2408885998500000006
105        
9                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

106        ;*';*";*/;
107        data _null_; e = exist('user._df');
108        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 0
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
109        
110        
111        ;*';*";*/;
112        %put E3969440A681A2408885998500000007;
E3969440A681A2408885998500000007
113        

df_sas.describe()

	Variable	N	NMiss	Median	Mean	StdDev	Min	P25	P50	P75	Max
0	speed	50	0	15	15.40	5.287644	4	12	15	19	25
1	dist	50	0	36	42.98	25.769377	2	26	36	56	120

print(session_sas.saslog())

1                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

NOTE: Copyright (c) 2002-2012 by SAS Institute Inc., Cary, NC, USA. 
NOTE: SAS (r) Proprietary Software 9.4 (TS1M1) 
      Licensed to LICENCE MESR SAS RECHERCHE PCW, Site 70109801.
NOTE: This session is executing on the W32_7PRO  platform.

NOTE: Updated analytical products:
      
      SAS/STAT 13.1
      SAS/ETS 13.1
      SAS/OR 13.1
      SAS/IML 13.1
      SAS/QC 13.1

NOTE: Additional host information:

 W32_7PRO WIN 6.1.7601 Service Pack 1 Workstation

NOTE: Initialisation de SAS used (Total process time):
      real time           0.01 secondes
      cpu time            0.01 secondes
      
1          ;*';*";*/;
2          options svgtitle='svgtitle'; options validvarname=any pagesize=max nosyntaxcheck; ods graphics on;
3          
4          ;*';*";*/;
5          %put E3969440A681A2408885998500000001;
E3969440A681A2408885998500000001
6          
2                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

7          ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
7        ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
8          ;*';*";*/;
9          libname work list;
NOTE: Libref=   WORK 
      Scope=    IOM ROOT COMP ENV
      Moteur=   V9
      Access=   TEMP
      Nom physique= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD2492_PMGD_\Prc2
      Nom du fichier= C:\Users\dondon\AppData\Local\Temp\SAS Temporary Files\_TD2492_PMGD_\Prc2
10         
11         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
12         
13         %put E3969440A681A2408885998500000002;
E3969440A681A2408885998500000002
14         
3                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

15         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
15       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
16         ;*';*";*/;
17         %put SYSV=&sysvlong4;
SYSV=9.04.01M1P12042013
18         
19         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
20         
21         %put E3969440A681A2408885998500000003;
E3969440A681A2408885998500000003
22         
4                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

23         ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
23       ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
24         ;*';*";*/;
25         proc options option=encoding;run;

    SAS (r) Proprietary Software Version 9.4  TS1M1

 ENCODING=WLATIN1  Specifies the default character-set encoding for the SAS session.
NOTE: PROCEDURE OPTIONS used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
26         
27         ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
28         
29         %put E3969440A681A2408885998500000004;
E3969440A681A2408885998500000004
30         
5                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

31         data _df;
32         length  'speed'n 8 'dist'n 8;
33         infile datalines delimiter='03'x DSD STOPOVER;
34         input @;
35         if _infile_ = '' then delete;
36         input 'speed'n 'dist'n ;
37         datalines4;

6                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

NOTE: DATA statement used (Total process time):
      real time           0.06 secondes
      cpu time            0.01 secondes
      
NOTE: The data set WORK._DF has 50 observations and 2 variables.

89         ;;;;
90         run;
91         
7                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

92         ;*';*";*/;
93         
94         
95         ;*';*";*/;
96         %put E3969440A681A2408885998500000005;
E3969440A681A2408885998500000005
97         
8                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

98         ;*';*";*/;
99         data _null_; e = exist('_df');
100        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 1
NOTE: DATA statement used (Total process time):
      real time           0.01 secondes
      cpu time            0.00 secondes
    
101        
102        
103        ;*';*";*/;
104        %put E3969440A681A2408885998500000006;
E3969440A681A2408885998500000006
105        
9                                                          Le Système SAS                        18:15 Wednesday, September 12, 2018

106        ;*';*";*/;
107        data _null_; e = exist('user._df');
108        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 0
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
109        
110        
111        ;*';*";*/;
112        %put E3969440A681A2408885998500000007;
E3969440A681A2408885998500000007
113        
10                                                         Le Système SAS                        18:15 Wednesday, September 12, 2018

114        ;*';*";*/;
115        data _null_; e = exist('WORK._df');
116        te='TABLE_EXISTS='; put te e;run;

TABLE_EXISTS= 1
NOTE: DATA statement used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
117        
118        
119        ;*';*";*/;
120        %put E3969440A681A2408885998500000008;
E3969440A681A2408885998500000008
121        
11                                                         Le Système SAS                        18:15 Wednesday, September 12, 2018

122        ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
122      ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
123        ;*';*";*/;
124        proc means data=WORK._df  stackodsoutput n nmiss median mean std min p25 p50 p75 max; ods output Summary=work._summary;
124      ! run;
NOTE: The data set WORK._SUMMARY has 2 observations and 11 variables.
NOTE: There were 50 observations read from the data set WORK._DF.
NOTE: PROCEDURE MEANS used (Total process time):
      real time           0.09 secondes
      cpu time            0.04 secondes

125        
126        ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
127        
128        %put E3969440A681A2408885998500000009;
E3969440A681A2408885998500000009
129        
12                                                         Le Système SAS                        18:15 Wednesday, September 12, 2018

130        ;*';*";*/;
131        proc sql;
131      !           create view sasdata2dataframe as select * from work._summary;
NOTE: SQL view WORK.SASDATA2DATAFRAME a été définie.
131      !                                                                        quit;
NOTE: PROCEDURE SQL used (Total process time):
      real time           0.04 secondes
      cpu time            0.01 secondes
      
132        data _null_; file LOG; d = open('sasdata2dataframe');
133        length var $256;
134        lrecl = attrn(d, 'LRECL'); nvars = attrn(d, 'NVARS');
135        lr='LRECL='; vn='VARNUMS='; vl='VARLIST='; vt='VARTYPE='; vf='VARFMT=';
136        put lr lrecl; put vn nvars; put vl;
137        do i = 1 to nvars; var = compress(varname(d, i), '00'x); put var; end;
138        put vt;
139        do i = 1 to nvars; var = vartype(d, i); put var; end;
140        run;

LRECL= 88
VARNUMS= 11
VARLIST=
Variable
N
NMiss
Median
Mean
StdDev
Min
P25
P50
P75
Max
VARTYPE=
C
N
N
N
N
N
N
N
N
N
N
NOTE: DATA statement used (Total process time):
      real time           0.04 secondes
      cpu time            0.01 secondes
      
141        
142        ;*';*";*/;
143        %put E3969440A681A2408885998500000011;
E3969440A681A2408885998500000011
144        
13                                                         Le Système SAS                        18:15 Wednesday, September 12, 2018

145        ;*';*";*/;
146        data _null_; set work._summary(obs=1 );put 'FMT_CATS=';
147        _tom = vformatn('Variable'n);put _tom;
148        _tom = vformatn('N'n);put _tom;
149        _tom = vformatn('NMiss'n);put _tom;
150        _tom = vformatn('Median'n);put _tom;
151        _tom = vformatn('Mean'n);put _tom;
152        _tom = vformatn('StdDev'n);put _tom;
153        _tom = vformatn('Min'n);put _tom;
154        _tom = vformatn('P25'n);put _tom;
155        _tom = vformatn('P50'n);put _tom;
156        _tom = vformatn('P75'n);put _tom;
157        _tom = vformatn('Max'n);put _tom;
158        run;

FMT_CATS=
$
BEST
BEST
D
D
D
D
D
D
D
D
NOTE: There were 1 observations read from the data set WORK._SUMMARY.
NOTE: DATA statement used (Total process time):
      real time           0.01 secondes
      cpu time            0.01 secondes
      
159        
160        ;*';*";*/;
161        %put E3969440A681A2408885998500000012;
E3969440A681A2408885998500000012
162        
16                                                         Le Système SAS                        18:15 Wednesday, September 12, 2018

170        ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
170      ! ods graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: _TOMODS1
171        ;*';*";*/;
172        proc delete data=work._summary; run;

NOTE: Deleting WORK._SUMMARY (memtype=DATA).
NOTE: PROCEDURE DELETE used (Total process time):
      real time           0.00 secondes
      cpu time            0.00 secondes
      
173        
174        ;*';*";*/;ods html5 (id=saspy_internal) close;ods listing;
175        
176        %put E3969440A681A2408885998500000013;
E3969440A681A2408885998500000013
177       

[tomweber-sas]

I think I figured it out! It is the translated ODS in the earlier SAS releases. They’re different then current releases.
I was going to have you try out one more thing, but while trying it myself, to be sure it would work for you,
I ran it against an earlier SAS version. The earlier versions have the ODS templates in session encoding, but
the more recent versions are always utf-8. That’s the only way I’ve ever seen them. Now I see that the ‘fix’
in the sasioiom.py at master is actually right. It tryss the utf-8, like is in the newer SAS, but backs down to
session encoding if that fails, which is what the old templates are. That’s ‘why’ it works.

Sweet. You’re done and should be fixed! No need to do any more testing for me.
I’ll build a new release with this fix so the other issues I’ve had on this are resolved with the latest saspy version. 2.2.8.

Thanks for helping me track this down!
Tom

[mgdondon]

Thank you for the explanation.

[tomweber-sas]

No problem. Kinda for everyone who might run into this too. It appears it was 9.4M3 that changed to be utf-8 all the time. Previous versions are session encoding.

I'll close this issue, but link it to the others where I've seen this same problem.

Thanks!
Tom

[tomweber-sas]

FYI:
I've pushed a notebook showing the discrepancy here:
https://github.com/sassoftware/saspy-examples/blob/master/Issue_examples/Issue168-Resolution.ipynb

[jurandifr]

Is the utf-8 decode bug fixed in the current version?
I get the same error.

[tomweber-sas]

@jurandi82 yes, this was fixed 3 years ago. Will you open a new issue with your information: what versions of saspy/SAS, the code you're submitting and the output it gets (all of it), your configuration info, and anything else you think would be relevant?
Thanks!
Tom

[jurandifr]

Saspy still ignore other encoding in config, I use pandas to read latin1 and upload as utf-8 and works for me.