sas2df doesn`t work with no response

[Tracyseu]

Server: Linux Centos6.7; SAS 9.4m4
Local: win; python 3.7; Jupyter
Connect mode: winiomlinux

Problem description:
When I use sd2df function to switch the sas data from server, python console is just running with response.
For example,

	df_test = sas.sd2df("sasdata")

I make sure the sas session is valid and sasdata is in work lib.
I also modify different methods, including 'MEMORY' and 'CSV', but it doesn`t work either.

Is it the problem from the connect mode, and how can I fix this?

[Tracyseu]

BTW, the the size of my dataset is so small , less than 1000 obs and 10 vars.

[Tracyseu]

I also try create dataset in dataframe firstly, then I use df2sd function to switch it into sasdata . Next, I use sd2df function to opearate the same dataset back to dataframe. Its successful. But it doesnt work with the dataset created in sas system.

[tomweber-sas]

Hey @Tracyseu I can help you with that. But I need some information. Can you submit the following and show the results? As you're in Jupyter, do each line or group below in a separate cell or we won't be able to see all of the output. If 'sasdata' isn't really the name, swap out the correct name in the code below, of course.

# after import saspy and sas=saspy.SASsession()
saspy

sas

sastab= sas.sasdata('sasdata', results='text')
sastab

sastab.contents()

sas.sascfg.name

cfgdef = getattr(sas.sascfg.SAScfg, sas.sascfg.name)
for parm in cfgdef:
   print(parm+' = '+cfgdef[parm])

df_test = sas.sd2df("sasdata")

Thanks!
Tom

[tomweber-sas]

Of course, you can obfuscate any of the output you need to, if there's anything in it that you don't want posted. But I need to see that info in general.

thanks,
Tom

[Tracyseu]

@tomweber-sas Thanks for your help. Because of the LAN environment, I will show you info in pictures. And in the last cell, kernel is still just running with nothing output.

[tomweber-sas]

@Tracyseu Thanks for this information! It's just what I needed. First I wanted to know what version of saspy you have, and if the saspyiom.jar what the right one to match the rest of saspy python code. Things like that, as I didn't see this problem here.

Everything looks correct with your configuration and you have the latest version. The saspyiom.jar is from the saspy install, as is your config file, all from the same dir:
C:\Anaconda3\lib\site-packages\saspy...

Your connection looks good ans is functioning and the data set looks like it should be no problem either, and you are running utf8 with a Chinese local. All of this looks fine.

I tried to reproduce this as best I can, and although it was all working for me, I did have one case where I saw, what seemed to be, the same problem. I can't be sure, as all I know is sd2df seems to have not returned for you, but I can't see why.

But, as I dug into this today, I do expect that we are seeing the same problem. I have found a bug in SAS, in the 9,40M5 version only. This works correctly at 940, M1, M2, M3, M4 and M6. But it gets into an infinite loop at M5 writing blanks to the output file and will run until it's killed (or the file can't be written to I suppose). You are running w/ SAS 940M5 so I expect this is the problem you are running into. But, I can't be 100% sure yet.

The bug is in the data step and has to do with multi-byte characters.

Can you verify that sd2df can read data sets that only have ASCII data in them? Do you have sashelp data sets that are not translated? Or, you can just create a data set with only A-Z and 0-9 character fields to try this out. Also, do you have any other versions of SAS than M5 that you could test this same case out with? I would like to verify that this is the same problem.

I will need to do something for you so that you can use saspy with M5; I'll have to work around this SAS bug in my saspy code somehow. I will work on that next, in case you only have M5 that you can use for what you are doing. It will be the beginning of next week as it's almost the weekend now. But I should be able to get you something else to try out so we can be sure this is the problem and I can work around it.

Sorry about this! But I'll get you up and running!

Thanks,
Tom

[Tracyseu]

Hi @tomweber-sas , my saspy`s version is 3.1.8.

I try to use the dataset named cars in sashelp, and run successfully with sd2df function. It seems it`s ok.

And I just only have 9.4m5 so I can`t test the same problem whether happens in other version environment or not.

Thanks for your help! This problem seems to be fixed soon.

[tomweber-sas]

Thanks for that. Yes, I believe you are hitting this bug in the data step. I'm working on a solution in saspy to work around it. I'm also trying to get a hotfix for this to fix SAS, (M5) but that won't help you for some time, if that even gets done. I'm close to having this worked out, but not 100% yet.
When I do have something, are you comfortable with pulling from master or another branch to test out the fix? If not, I can work you through that. I am hoping to have this ready to try out tomorrow.

Thanks,
Tom

[Tracyseu]

Hi @tomweber-sas , I`m glad to help you to test the fix. I will be expecting.

[tomweber-sas]

thanks @Tracyseu . I have a test fix that I've pushed to a branch named 'm5bug'. Can you pull that and try it? It is not the final fix, but it should prove if the problem you are seeing is the same as what I found. And, it should then get you working correctly. But, that is to be seen :)
As we are 12 hours off, I was hoping that you might be able to verify, or not, this for you today, then when I get in in the morning, I can get the final solution (tomorrow night your time), so I can have the real answer for you on your tomorrow; as opposed to the day after. I know it makes things slow being asynchronous like this.

Thanks!
Tom

[Tracyseu]

Hi @tomweber-sas, the problem has fixed with the new code. Thanks so much!

[tomweber-sas]

Hey, great! You can use that for the mean time. I will work on putting together a 'final' fix for this. The thing is, the code that's broken at M5 is what it should be, especially for the _DISK method. So I will probably end up with an option, for the case where you have M5 without the hotfix (if/when that's available), to generate different code (this fix) that won't necessarily be as performant. So, I expect I'll default to the better code, with the M5 bug, by default, and require an option set so the code generates the 'safe' code (again, this fix) for the situation with the bug in V9.40M5 SAS.

I should have this all figured out and pushed for you tomorrow to try out, so I expect you will be able to try it tomorrow (for you). Headed to bed now :)

Thanks again!
Tom

[tomweber-sas]

Ok, I've updated the m5bug branch with my latest version. The code generation is the same as what you tried, but I've added an option, 'm5dsbug' that can be set in the configuration definition, or of course, on the SASsession() method to choose what code is generated. I am checking the SAS version that saspy is connected to to see if it's one of the versions affected by this bug: v940M5, but also SPRE in Viya versions 3.3 and 3.4. So, if I see I'm connected to any of those, I'll issue a note about this bug and that I will use the alternate code to work around this bug. If you set the 'm5dsbug' option, then I just honor it without any messages.
So, you can try this out and see that you get the message and that it still works correctly. Then, you can add the option to your configuration definition, so you no longer get that message, and it still works:

winiomlinux = { ...,
              'm5dsbug' : True,
               ...,
               }

There is no hotfix for this bug, but if there ever is one and you were to have it applied, then you would set m5dsbug=False and I would then use the code gen I wanted to and it would work, like in the other versions. But, no hotfix as of yet.

If you can try this out and make sure I haven't regressed anything, I'd appreciate it. I'll continue to test and validate this so I can see about moving it to production.

Thanks again!
Tom

[Tracyseu]

Hi @tomweber-sas , I renew then saspy package and add the option as you list in my config.
However, when I use sd2df function, it throws out an error like this:

• AttributeError: 'SASsession' object has no attribute 'm5dsbug'.

It happens in 'saspy\sasioiom.py' , sasdata2dataframe, line 1600.

[tomweber-sas]

Hey, are you still online? Just got home from playing hockey. I think I see that I missed a case. It worked when you didn't set it, and you got the message? I see I need an else: and set it in one spot. I just pushed that, so I think this should be fixed now if you reinstall from m5bug. Sorry about that!

[Tracyseu]

Hi @tomweber-sas , the latest version is ok!

[tomweber-sas]

Great, thanks! I still have some investigation to do to verify this bug is in the viya deployments that it appears to be in. Don't have them directly available to test out. So, I need to do a bit more before making this production. But, you can use this version for now until I get this all the way to production. Thanks again for your help testing this out!

Tom