Dead end in sas = saspy.SASsession()

[ghost]

Hi, I'm trying to install the SAS kernel on jupyter. Currently I can connect to the kernel, but no matter what I submit to run, there is a dead end with no response. I tried to run

sas = saspy.SASsession() (or with cfgname = 'winlocal')

and it is a dead end as well. I've included the path of the .dll in my system environment variable PATH.

Thanks for your help!

[tomweber-sas]

Sorry to hear that. I can help. Can you show me what you're seeing? what version of saspy did you install? How did you install it? What version of SAS is installed on the machine you're trying to run this on? What's in your config file? What do you actually get when you try to connect?
Thanks,
Tom

[ghost]

Hi Tom,

glad to have your help. I install saspy by 'pip install saspy', and version is 2.2.7

info for SAS:

related parts of config file:

I'm trying to work on my own windows machine.

Thanks!

[ghost]

I'm not sure if I should uncomment the last three parts of cpW

[tomweber-sas]

Ah, good, that' spart of what I need. It looks as though you don't have '\' in the classpath entry for the saspyiom.jar, after \java. Can you double check that? So what error are you seeing when you try to connect? That should have put out messages about not finding the jar and a bunch of stuff.
Can you run this in a python notebook and show me the results?

import saspy
sas = saspy.SASsession(cfgname='winlocal')

Correct any typos in there if there are any; I'm freehand typing that. ")

[tomweber-sas]

I wouldn't worry about those last 3 for now. I need to see what error you're getting first.

[tomweber-sas]

And, I just built 2.2.7 and I haven't had a chance to pip install it fresh myself, so let me do that on my windows box to be sure that's all good.

[tomweber-sas]

ok, I installed 2.2.7 on my pc and it's good, so let's fix your classpath and see what you're getting.

[ghost]

Oh yes! It was a typo in copy/paste I guess.

Now I make some progress, and the output in python is

but when I run in jupyter, I get

The dataset is created I think, with 1 obs and 3 vars. However, the second piece of code seems to be a dead end...

Thanks!

[tomweber-sas]

Are you running all of that in one notebook? I'm not sure what I'm seeing there. I Jupyter there are different kinds of notebooks; different kernels. They each process a different language. saspy is used in the python kernel, as it's python language. If you're in the sas_kernel notebook, you can only type in sas code, but the sas_kernel just submits that to saspy under the covers.

Are you running the 'sas = saspy.SASsession()' stuff in python kernel? and what about the data step code? in the sas_kernel?

[ghost]

I ran sas = ... in python kernel, and those data and proc steps were written in jupyter with a sas kernel

[tomweber-sas]

ok, so let's do something in python to see if it's all working right. It seems to be connecting, but the workpath seems odd. Can you run the following in python; put each statement that generates output in a different cell, because each cell only shows 1 output (the last).

import saspy
sas = saspy.SASsession(cfgname='winlocal')
sas

#new cell
cars = sas.sasdata('cars', libref='sashelp')
cars.head()

#new cell
print(sas.saslog())

Thanks,
Tom

[ghost]

Sure, I got

for the first two, and the third output is too long. Do you need all those information in the third one? It's hard to take a screenshot...

[tomweber-sas]

actually, no. That shows it's working for you. So, can you just submit the following in a sas_kernel notebook?

proc print data=sashelp.cars(obs=5);run;

That should be the same as what we just did in python.

[ghost]

I got

and sadly nothing else.

[tomweber-sas]

Oh, so btw, you're running SAS in Chinese. But you are telling python you're running it in Wlatin1; the 'encoding' value in your configuration file for winlocal. Can you change that to the following in your sascfg[_personal].py:

'encoding' : 'gb2312'

I'm not sure that this would account for the sas_kernel to hang, but maybe? It's certainly wrong for this case.

[tomweber-sas]

You'll need to restart the notebooks for the change to your config file to take effect. So, restart the kernel is the thing to do in the notebooks

[ghost]

I will try that now. btw, I usually use the English version of SAS, but it seems that when I run sas.exe in SASFoundation\9.4, it always runs in Chinese version. Is that because the system language is Chinese? Is there a way to open up SAS in English version without changing the system language?

Thanks

[tomweber-sas]

SAS has a sasv9.cfg file under SASHOME\SASFoundation\9.4 directory. That's the config file where you can configure things like that. I think that file points to one in the nls directory and then that's the one it uses. When you installed it, it probably picked up Chinese from your system, but you can set it by specifying these in the sasv9.cfg instead:
-LOCALE en_US
-ENCODING wlatin1

Did matching up the encoding help sas_kernel?

[ghost]

I guess there is some problem with encoding. I've changed the value associated with encoding, and restarted the notebook, but it's still not working.

I got

Thanks.

[tomweber-sas]

ok, was that with the encodings matching, or what you got running Chinese but telling saspy it's windows 1252?

[ghost]

with encoding = gb2312 I think

[ghost]

Yeah, I think it's the language issue

[tomweber-sas]

It would be good to see the log so I can tell what's going on. Can you copy the saslog() output from the saspy run to a file and just email it to me? Can you do it for both cases w/ encoding of windows-1252 and gb2312? I don't know why we would get this error through the sas_kernel (in saspy) but not with saspy directly.

[ghost]

after I modified the sasv9.cfg to

-config "D:\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg"

the language in SAS has changed to EN, and then I get the output

with gb2312

[ghost]

However, the output seems to be not well aligned, I mean, lots of overlaps. Is there a way to fix that?

[tomweber-sas]

Ok, that's good, but 2.2.7 has some fixes that make it work very cleanly w/ other SAS encodings. So I'm still curious about that. However, the data sets may not be in Chinese, so maybe there's a mismatch with what SAS is running , and what it thinks it has and what bytes are actually coming down the pipe. I'm not sure why it would work with saspy directly but not with saspy called by sas_kernel.

Yes, that overlap problem is a bug in the jupyter notebook. I can't fix that, but Jupyter has a newer version called JupyterLabs where they've fixed that bug. Earlier versions of Notebook also work fine too.

[ghost]

Yeah, now I've changed encoding back to windows-... and I think it should be fine to work with. Thanks for your kind and patient help! I've been working on this the whole day. If you want me to try something that what you are curious about, I'd like to help.

[tomweber-sas]

Great, thanks. I'm logging off for today, but I may give you a couple things to try just to see what we see.
I've done a lot of work to fix up transocding problems in this new release. So it could be there's some other side effect of having your OS language be Chinese (which I haven't/can't try) that could be doing something. Like Java running in Chinese instead of utf-8 or something. Could be some other path I've not seen before.
Glad you're working for now though!
Thanks,
Tom

[ghost]

Thanks for your help, and I'll try to help. The issue is some people like me are using the Win10 home version in which the OS language cannot be changed...

[tomweber-sas]

@afslekj Thanks for offering to help! I have a notebook that you can run 3 different ways. Once with each of the following encodings to see what we see. You will need to change your sasv9.cfg and sascfg_personal.py for each run, to match up the encodings correctly. The first cell shows the sas/saspy encoding values, which are Windows Latin1, utf-8 and Chinese.
If you can run these and then just download them as html (that will have all of the output in it) and email those html files to me, I can see what the logs show. If you get a failure in a cell, just record that and then restart and skip that cell and see if the others run.
You can see that you've got the encodings matched up right betwee SAS and saspy by looking at the output from submitting the SASsession object (the encodings are at the bottom):

SAS Connection established. Subprocess id is 8876

Access Method         = IOM
SAS Config name       = winlocal
WORK Path             = C:\Users\sastpw\AppData\Local\Temp\SAS Temporary Files\_TD13672_d10a626_\Prc2\
SAS Version           = 9.04.01M4P11092016
SASPy Version         = 2.2.7
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = WLATIN1
Python Encoding value = cp1252

Here's the notebook. Save it and remove the .txt. Then you can upload it into Jupyter and run it.
I'm, unfortunately, going to be unavailable the rest of this week, but if you can try this out and see what happens, I'll follow up on it next week.

Thanks in advance!
Tom

Issue157.ipynb.txt

[ghost]

Sure, I see. I'll try what you said. :-)

[ghost]

I've sent those results to your .sas email. Hope it is helpful :-)

[tomweber-sas]

@afslekj Just to let you know, I did receive that email. I'm still trying to catch up from being out last week, but I have them and will hopefully dig in starting tomorrow.
Thanks!
Tom

[tomweber-sas]

@afslekj a quick update. I see 'the difference' between your system and mine. And I will need to investigate this further. When I run these tests here, I have Chinese data, UTF-8 (Chinese) data and all of the transcoding works with SAS running these different encodings. BUT, the difference I see is in our SAS LOGs. My logs are still in english, other than any data specific parts that may be written there. However, your logs have many parts in Chinese. So, you have a 'translated SAS system' which isn't something I've dealt with before. That's why the

WORK Path     = /

which I was wondering about. I parse the LOG to get that, but that part of the log is Chinese, so I'm not getting what I expect. I see an error for the sd2df_CSV case, which I can't read, that seems to be the reason for failure of that case.

ERROR: 物理文件“C:\Users\张一凡\AppData\Local\Temp\tmppwwg98dk\tomodsx”不存在。

Can you tell me what that says?

I suspect that I have issues w/ translated versions of SAS that I'm going to have to figure out. I can get up with our NLS division and try to track down the differences that I will have to be able to handle.

These logs will certainly help me with that. I'll see if there is something I can do in the short term to handle the specific errors you're hitting. Though there may be other situations where I get tangled up given the translated version.

Tom

[ghost]

Hi, that error basically says

The file c:... does not exist

[tomweber-sas]

Ok, thanks!
I talked w/ the NLS division and they told me how I should be able to run SAS to match how you're running it; translated. Obviously, my OS language will still be english, but I'm not seeing anything so far to make me thing the OS lang has anything to do with this. So, hopefully I'll be able to reproduce these and see how to work around them. It'll be tomorrow at best before I can try that though. Multitasking too much today.
For the failure with that error; the line: dfc = carst.to_df_CSV()
Can you run that with these options, specifying a valid filename just to see if it works? I suspect something with the characters in the path had something to do with that file not being created.
tempfile: str = None, tempkeep: bool = False,

dfc = carst.to_df_CSV(tempfile="D:\SASHome\SAS Temporary Files\_ods.csv", tempkeep=True)

Just have whatever path you choose only contain ASCII characters. This may either work, or still fail perhaps, but if so we will have the file there to see what the contents are. Again, I may be able to reproduce this here, hopefully.

Thanks again,
Tom

[ghost]

Yeah, that username has caused lots of trouble. I'm trying to modify it. I'll try it out, but perhaps tomorrow ～

[ghost]

@tomweber-sas I sent you an email regarding your last comment.

[tomweber-sas]

Got it, thanks! So, that worked, so I'll have to figure out what the problem actually was with the user name. I'm not sure about that yet. The filename statement didn't show an error (or anything at all), so I'm not clear what went wrong there.
There was no problem with to_df_CSV(), though there was no non-ASCII data to retrieve, but even if there was, that should be ok. I'm trying to reproduce the error with the HTML version of head() though. I don't yet know why that would be getting a transcode error. But, in the translated versions of SAS, it not only the message files that are translated, but the ODS templates too, so that bound to be part of it.
I've been able to run SAS on Linux with -LANG zh; which is closer to your scenario, as I now see much of the log in Chinese, whereas only setting -encoding EUC-CN didn't use the translated message files or ODS templates.
Unfortunately, I didn't see a transcoding error which you are getting for the HTML case. I'm having to reinstall SAS on my pc to install the Chinese version so I can try this on Windows. I'm hoping I can see this problem then, but we'll see.
As for the issue w/ 'workpath=' for this, at least that's not causing any problems, as I'm not using that value for any code I generate. So, even though that's wrong, it's not being a problem.

So, right now, it's the ODS problem that I'm trying to reproduce and track down. I'll also investigate the user name problem if I can.

More to come,
Tom

[tomweber-sas]

@afslekj So I think I've finally tracked this down. I was able to reinstall SAS w/ Languages on my pc and when I ran pointint to the ZN configuration (running Chinese) and ran in DMS, the ODS output wasn't written as utf-8, but rather in Chinese encoding. When I ran that same SAS via local IOM to saspy, I still got the ODS output in utf-8, not Chinese. That's why I've not been able to reproduce this on my side.

But, on your system, I bet you're getting the CHinese encoding back to saspy, not utf-8. It'll take me more time to figure out how/why this is. But, I think I have a fix for this, which I've pushed to Master.

Can you grab the code at master and run that test notebook to see if your HTML output now works?
This should be the problem with sas_kernel too, as it is only HTML output.

I have an HTML output of the notebook I used to track this down:
issue157_a.html.txt
Just remove the .txt (can't upload .html).

You'll see the same error you were getting and that if I transcode using the SAS encoding it comes out fine.

The fix I pushed is just basically to do that; try transcoding via utf8, which it right for all the usual case. If that fails, try using the encoding value, and in case that gets a transcoding error, transcode with any error's replaced with the substitute char so that at least it will come through.

See if this works for your cases, please. This still wouldn't address the username in the path for CSV. I haven't had a chance to look into that.

Thanks,
Tom

[ghost]

Hi Tom,

I will take a look, probably this weekend. I've generated a new user account with English username, I'll run these in that account.

[ghost]

Hi Tom,

I will take a look, probably this weekend. I've generated a new user account with English username, I'll run these in that account.

[tomweber-sas]

Do you happen to have changed the following option in your sascfg file? Setting this to something other than HTML5? The case where I was getting the html back in Chinese encoding (in my DMS session) the ODS setting wasn't HTML5. When I changed it to HTML5 (it was using html4) I did get utf-8 back and there was no problem.

When I set the saspy config option to use html4, I saw the same error you are getting, as the html document was not encoded chinese, not utf8.

I just can't get your error through saspy with anything I try, except for when I'm not using html5.

# 'output' = ['html5', 'html']
#
SAS_output_options = {'output' : 'html5'}

Thanks!
Tom

[tomweber-sas]

One other thing. I expect the change at master will keep you from getting an error. However, I'd still like to see the html document you are getting. If you can run the following code and send me the output;, that will help a lot.

sasz.batch=True
x = cars.head()['LST']
sasz.batch=False
print(x)

Thanks again!
Tom

[ghost]

No it's Html5...

[ghost]

What config do you want when I run those codes?

[tomweber-sas]

Well, ideally the same as what you ran before. Chinese and English. The Chinese case is the most curious as I can't make it fair other than by not using html5.
I'm working with our NLS division to try to understand what factors go in to what html template and encoding is used.
Thus the question about that option. Though if you can send me the document you get back that will hopefully clear that up.

[ghost]

Okay, I'll test these as well this weekend ~

[ghost]

Hi Tom,

I tried to run your notebook with the new version in master with Chinese SAS and utf-8, cp1252 encoding for saspy. Both of them ran without error, but some Chinese characters are transcoded to non-sense symbols. As for
sasz.batch=True x = cars.head()['LST'] sasz.batch=False print(x)
I don't know what sasz and cars are, and they are not recognized by compiler. I have no experience in coding with saspy before, so I don't know these syntax.

[tomweber-sas]

Oh, I'm sorry. My bad. I just cut-n-paste that out of something I was running. I should have given you the whole thing. Yes, I'm glad it ran w/out error, but I'm trying to figure out what encoding the html is coming across as. I can't make it do what you're seeing for the cases I know of. That's why I wanted to see what's in the document.
This should be easier for you to use:

import saspy
sas = saspy.SASsession(cfgname='winlocal')
cars = sas.sasdata('cars', 'sashelp', results='html')
sas.batch=True
x = cars.head()['LST']
sas.batch=False
print(x)
# different cell if in jupyter, so you get the output
print(x.encode())

[tomweber-sas]

@afslekj I believe I've finally tracked down the different between your case and mine: #168
You are running v94M2 and I've on;y been on M3 or above. Even though I can run with lang and encoding and everything in Chinese, getting Chinese in the log like you , my ODS template was UTF-8. While yours is GBK (EUC_CN, ZH, ...) Chinese. I see that the translated ODS templates pre-M3 were in the session encoding, and M3 and beyond are utf-8.
So what I was seeing w/ the HTML4, was the same problem, even w/ HTML5 in the preM3 versions.

This means that the code at Master, which I will promote to 2.2.8, will fix this correctly. It tries to transcode via utf8 first, but uses SAS session encoding if that fails.

You can ignore the last request for output as I can finally reproduce it here and see the issue.
If you can use Master, grab it and see that it resolves the problem. I'll be promoting that to 2.2.8 as soon as I can.

Thanks!
Tom

[ghost]

Hi Tom,

Great to hear that. However, I've been busy these days so I really cannot spare much time on this. I'll update it to 2.2.8 once you release it. Looking forward to the new version!

[ghost]

Hi Tom,

Great to hear that. However, I've been busy these days so I really cannot spare much time on this. I'll update it to 2.2.8 once you release it. Looking forward to the new version!

[tomweber-sas]

No problem, I know how it is :) I'll let you know when I get that out there!

[tomweber-sas]

FYI, 2.2.8 is available :)

[ghost]

Great!! Thx Tom