Is there any way to browse the sas server directories for browsing files and folders in saspy (for user interface)? How to get SAS server directory tree in SASpy?

[mailbagrahul]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[tomweber-sas]

Well, no, I don't have any methods for browsing the file system of the SAS server that saspy is connected to. To be access method agnostic, it would need to be SAS code to be submitted to access the file system. SAS itself doesn't have a terribly east interface to use to do this. Looking at functions for this, there are data step functions, though they appear very tedious to try to use to implement a file system browser api. https://go.documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.2&docsetId=lefunctionsref&docsetTarget=p1bhwd0nlni4own1ia79mj0n29hv.htm&locale=en

Here's just an example to get basic current directory info.

 79? data _null_;
   length opt $100 optval $100;
   /* Allocate directory */
   rc=FILENAME('mydir', '.');
   /* Open directory */
   dirid=DOPEN('mydir');
   /* Get number of information items */
   infocnt=DOPTNUM(dirid);
   /* Retrieve information items and */
   /* print to log                   */
   put @1 'Information for a UNIX
      File System Directory:';
   do j=1 to infocnt;
      opt=DOPTNAME(dirid,j);
      optval=DINFO(dirid,upcase(opt));
      put @1 opt @20 optval;
 93?    end;
 94?    /* Close the directory */
 97?    rc=DCLOSE(dirid);
 98?    /* Deallocate the directory */
 99?    rc=FILENAME('mydir');
100? run;

Information for a UNIX      File System Directory:
Directory          /opt/tom/github/saspy/saspy
Owner Name         myuserid
Group Name         mygroup
Access Permission  drwxr-xr-x
Last Modified      05Nov2018:16:42:16
NOTE: DATA statement used (Total process time):
      real time           0.95 seconds
      cpu time            0.01 seconds

An alternative would be to use the X command, though that is frequently disabled on remote production servers. If you can use it, you can simply issue OS terminal commands to browse the FS and parse the info out of the log.

103? x ls -F1;
autocfg.py*
doc/
__init__.py
java/
libname_gen.sas
__pycache__/
sasbase.py
sascfg_personal.py
sascfg.py
sasets.py
sasiohttp.py
sasioiom.py
sasiostdio.py
SASLogLexer.py
sas_magic.py
sasml.py
sasproccommons.py
sasqc.py
sasresults.py
sasstat.py
sastabulate.py
sasutil.py
sasViyaML.py
tests/
titanic.csv
version.py
104?

I doubt either of these are what you were hoping for. But, what are your thoughts? How were you looking to use this? Any interest in implementing this functionality? For just a simple directory listing method, I suppose I could explore using the datastep and those functions to produce that. Seems clunky, but should be able to work.

Thanks,
Tom

[mailbagrahul]

That's my last option using data step to get the directory tree structure. Just wanted to check if something is available for user interface. I would need to do some crazy stuffs to have end user to browser the datasets in python web page..

[tomweber-sas]

Yes, I'm currently trying to use some of these functions to just get a directory list. Our examples in our doc don't help that much, unfortunately. But, I'll cobble something together and provide a prototype of this, so you can see if you think it would be helpful, or at least a start of what you're looking for.

[mailbagrahul]

Sure. That helps.

[tomweber-sas]

ok, here's something to poke at. I appended the os separator to the directories to distinguish from files. I could return a dictionary with dir or file instead. Depends upon what you want and how'd you use this,

>>> dl = sas.dirlist('/opt/tom/github/saspy/saspy')
>>> for i in dl:
...  print(i)
...
__init__.py
sasbase.py.bak
sasproccommons.py
sasbase.py
sasstat.py
sascfg_personal.py
sasiohttp.py
titanic.csv
SASLogLexer.py
sasqc.py
sasViyaML.py
sasutil.py
sasml.py
sasets.py
libname_gen.sas
sasresults.py
sasiostdio.py
__pycache__/
sastabulate.py
doc/
autocfg.py
sasioiom.py
version.py
sas_magic.py
sascfg.py
tests/
java/
>>>

Here's a little helper for separating dirs and files:

def dir_files(path):
   dl = sas.dirlist(path)
   print("Directories under "+path)
   for i in dl:
     if i[len(i)-1:] == '/':
        print(i)
   print("\nFiles under "+path)
   for i in dl:
     if i[len(i)-1:] != '/':
        print(i)

>>>
>>> dir_files(path)
Directories under /opt/tom/github/saspy/saspy
__pycache__/
doc/
tests/
java/

Files under /opt/tom/github/saspy/saspy
__init__.py
sasbase.py.bak
sasproccommons.py
sasbase.py
sasstat.py
sascfg_personal.py
sasiohttp.py
titanic.csv
SASLogLexer.py
sasqc.py
sasViyaML.py
sasutil.py
sasml.py
sasets.py
libname_gen.sas
sasresults.py
sasiostdio.py
sastabulate.py
autocfg.py
sasioiom.py
version.py
sas_magic.py
sascfg.py
>>>

So, what are you thinking with this. What functionality would you like more than just this? What format would you like, list, dict, ...?

What all methods to do some of this? Is this even in the right direction?

Tom

[tomweber-sas]

Here's a run from Windows. This won't work right on MVS though. I'd have to do something different for that. I assume you're not wanting this for MVS, right? :)

C:\Users\sastpw>python
Python 3.6.0 |Anaconda 4.3.0 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import saspy
>>> sas = saspy.SASsession()
Please enter the name of the SAS Config you wish to run. Available Configs are: ['default', 'SASgrid', 'http', 'httptest', 'ssh', 'httpfred', 'grid', 'tdi', 'iomj', 'iomc', 'iomjwi
n', 'winiomj', 'winiomjwin', 'winlocal', 'gridiom', 'wingridiom', 'zos', 'zos2', 'winzos', 'winzos2', 'winiomIWA', 'pune', 'notpune', 'issue176'] winlocal
SAS Connection established. Subprocess id is 796

>>> path = 'C:\ProgramData\Anaconda3\Lib\site-packages\saspy'
>>> def dir_files(path):
...    import os
...    dl = sas.dirlist(path)
...    print("Directories under "+path)
...    for i in dl:
...      if i[len(i)-1:] == os.sep:
...         print(i)
...    print("\nFiles under "+path)
...    for i in dl:
...      if i[len(i)-1:] != os.sep:
...         print(i)
...
>>> dl = sas.dirlist(path)
>>> for i in dl:
...   print(i)
...
__pycache__\
__init__.py
version.py
sas_magic.py
sasViyaML.py
sasutil.py
sastabulate.py
sasstat.py
sasresults.py
sasqc.py
sasproccommons.py
sasml.py
SASLogLexer.py
sasiostdio.py
sasioiomODS.py
sasioiom.py
sasets.py
sascfg_personal.py
sascfg.py
sasbase.py
libname_gen.sas
java\
autocfg.py
>>>
>>> dir_files(path)
Directories under C:\ProgramData\Anaconda3\Lib\site-packages\saspy
__pycache__\
java\

Files under C:\ProgramData\Anaconda3\Lib\site-packages\saspy
__init__.py
version.py
sas_magic.py
sasViyaML.py
sasutil.py
sastabulate.py
sasstat.py
sasresults.py
sasqc.py
sasproccommons.py
sasml.py
SASLogLexer.py
sasiostdio.py
sasioiomODS.py
sasioiom.py
sasets.py
sascfg_personal.py
sascfg.py
sasbase.py
libname_gen.sas
autocfg.py
>>>

[mailbagrahul]

Exactly.. This is what I expecting from the SAS directory(If pass some path to saspy. Yes, I do not want it for MVS. List should be fine i guess. Our next step would be making this list of files/folders to tree structure in python.

[tomweber-sas]

Is building some kind of tree structure something you would do with this method? Or are you looking for saspy to provide some further functionality to do that? Need other helper methods?

What are you looking for this to be (data structure) or look like (just a printed hierarchy)? Is it to be a traversable object or just something to display? Like 'tree' in DOS:

├───comtypes-1.1.2-py3.6.egg-info
├───conda
│ ├───base
│ │ └───__pycache__
│ ├───cli
│ │ └───__pycache__
│ ├───common
│ │ └───__pycache__
│ ├───core
│ │ └───__pycache__
│ ├───gateways
│ │ ├───adapters
│ │ │ └───__pycache__
│ │ ├───disk
│ │ │ └───__pycache__
│ │ └───^C
C:\ProgramData\Anaconda3\Lib\site-packages>
C:\ProgramData\Anaconda3\Lib\site-packages>cd saspy

C:\ProgramData\Anaconda3\Lib\site-packages\saspy>tree
Folder PATH listing for volume Windows
Volume serial number is E862-9DB5
C:.
├───java
│ └───pyiom
└───__pycache__

Thanks,
Tom

[mailbagrahul]

if saspy provide a way to choose a file from above tree (just like prompting) it would be easy for users

[tomweber-sas]

Can you show me what that would look like? Pssudo code with example output? I'm not completely sure I understand. If you choose a file, what happens?

[mailbagrahul]

something like this

[tomweber-sas]

So, you're implementing this UI, and would populate it with the list of files returned from calling sas.dirlist()?
Then a user would choose a file, then you would do what with it, from saspy? Have it transferred over into python? Assigned as a fileref?

[mailbagrahul]

Yes. you are right.

[mailbagrahul]

After the choosing the file, I will run thru sas.submit and render the output in python web page.

[tomweber-sas]

Ok, so what else do you need? You start with some path and call sas.dirpath(). They click on a file or directory. If dir, then you run that through dirpath() and display those results. So you can navigate up and down already.
When they choose a file, you submit code to assign it and read it, or is that my next method to add - transfer it over? sas.readfile(path) which returns what, a binary stream? Is this just for text files? Do you do something based upon the file extension?

[mailbagrahul]

if it is dataset, it should return dataframe or if it is .sas, execute the code and return the log. if it is excel, open the file using local machine's excel.exe

[tomweber-sas]

@mailbagrahul I've pushed this dirlist() to master so you can try it out and see how it works for you.
Let me know how it goes and if there's anything else you need.
This sounds like a cool project you're doing with the UI over saspy. I'm interested to see how it turns out!

Thanks,
Tom

[tomweber-sas]

I think you posted on the wrong issue. Did you get the code from master, or just do
pip install saspy?
I haven't promoted this to pypi yet.

[tomweber-sas]

you can install from master this way:
pip install git+https://git@github.com/sassoftware/saspy.git@master

[tomweber-sas]

If you're configuration's in your sascfg_personal.py, like it ought to be, you can just do the following:

pip uninstall saspy
pip install git+https://git@github.com/sassoftware/saspy.git@master

and you should be good to go.
Oh, unless you're on windows, which I suppose you are. Then download the zip from github (saspy page).
unzip it to a directory. cd to that directory then do

pip uninstall saspy
pip install .

Tom

[tomweber-sas]

Or you can just copy sasbase.py from master into the install you have. Any of the above.

[mailbagrahul]

Oops. I updated the comments in the wrong window.

[mailbagrahul]

Awesome. I got the list of files from the path given. Thanks a ton.!!!

[mailbagrahul]

Is there a way, i get the pid ? so that I can open the subprocess until the user logout.

[tomweber-sas]

Which pid are you looking for? The SAS server pid?

[mailbagrahul]

I mean the sas session id. I can open the session when the user signs in and disconnect after signs out.

[tomweber-sas]

The SAS pid is available off the SASsession object in the code you pulled from master.

sas = saspy.SASsession()
sas
sas.SASpid

would show:

>>> sas
Access Method         = IOM
SAS Config name       = winlocal
WORK Path             = C:\Users\sastpw\AppData\Local\Temp\SAS Temporary Files\_TD13912_d10a626_\Prc2\
SAS Version           = 9.04.01M5P09132017
SASPy Version         = 2.2.9
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = WLATIN1
Python Encoding value = cp1252
SAS process Pid value = 13912


>>> sas.SASpid
13912
>>>

Is that the id you're looking for? The OS process id of the SAS process itself?

[mailbagrahul]

I believe that may be the session id. Let me try to open two session at once and do some stuffs.

[mailbagrahul]

Ok. I did a small testing on pid. Here is the result. I guess you are not emptying out the session variables once it is disconnected. Is there any other variable that I can use it for validation?

[mailbagrahul]

So, What I did was. I opened SASsession and killed it explicitly and then i tried above statements. I used sas.disconnect() to terminate the session, is that not the right way to end the session? From your comment I understand that sas._endsas would terminate the session and clear out all the variables to None, Am I correct?

[tomweber-sas]

disconnect() is something different.
_endsas() is the method to end you session.

[mailbagrahul]

ahh. Just read the disconnect topic - https://sassoftware.github.io/saspy/advanced-topics.html#disconnecting-from-an-iom-session-and-reconnecting-back-to-it
Useful stuff. I will keep this in mind and implement.

[mailbagrahul]

Hi Tom,
How to make the SAS session open until I stop it manually. It is killing by itself after 5 seconds if I run the script thru command prompt but in Jupyter it is not the same case.

[tomweber-sas]

Well the SASsession will stay open until you close it,(_endsas()) or untill the SASsession object is deleted, or the python process terminates.
The message about not shutting down so killing it, is from the termination routine which would be driven by any of the above cases. I can't tell what is happening in the screenshot above. But, something is driving my saspy termination routine.
It looks like maybe you are making more than one connection? If you reuse the same object name, then you would be at the mercy of the garbage collector for determining when the original SASsession object would get deleted. And that could be different then when running in jupyter than batch. Just a guess, because I can't tell what's happening in that coed. I don't see any saspy code to tell what's happening.

[mailbagrahul]

Ok. I'm running thru local host by flask to connect saspy. I guess I'm not explicitly deleting the object or terminating the process. Let me find out whats happening at the background

[mailbagrahul]

Should I use set_batch() for scripts that runs thru cmd?

[tomweber-sas]

set_batch(True) will change the behavior of methods which return results. Instead of displaying the results (tables, grapsh, plots ....) they will return a dictionary, like the submit() method, with the result and the log. That way you can programatically interact with the result.
So, it depends on what code you're actually executing and what it's doing. Also on the mode of the output results (HTML, Pandas, Text). HTML results would need to be batch in this mode since that is an HTML document which won't 'render' in line mode or a batch python script. Text works file for tables, and so does pandas. But plots and graphs created by SAS will only be available as HTML.
So, again, you need to see what mathods you're running and what you're doing with the results to really know what to do. You can toggle batch() on and off for specific methods too, so it's not all or none.

[mailbagrahul]

Hi Tom - I have 2 questions for you.

1.Is there a way to retrieve all the datasets from the specified library from sas to dataframe?

One of my process runs a sas code that delivers more than 1 dataset that I would like to visualize in frontend using DASH package. I can do it for individual dataset. Its time taking for each request for sas server and fetch the data.

2. I have 100's of excel workbook stored in sas server. I have a code to scan and retrieve the excel documents names with path and store in dataset. I can pull the excel list and move to dropdown list but how can I download the workbook when user selects from the list?

Any thoughts?

[tomweber-sas]

There isn't a method to pull every data set in a library over to a data frame, but that wouldn't be too hard to implement. I also don't have a method to copy a file from the remote server back to the client; I wondered what you were going to do there a while back. I do have another issue (#187) asking for a file transfer method, so I was going to look into a generic file upload/download functionality. But I haven't started on that yet.
So, with the excel files, are you looking to copy the excel file to the client or load it into a SAS Data Set and bring it over as a data frame?

[mailbagrahul]

Great thanks. Pulling datasets to dataframes is the main focus for now. I'm making recursive request to server to pull the data. If it is not hard for you to implement, that would be great for anyone using saspy.

For file transfer, I want to download the files, render it on html page, make updates on the page itself and save it back to server.

[tomweber-sas]

ok, I've put something together for this. Want to run it by you to get some feedback.

I defined a method which, given a libref, returns a dataframe with the member names and memtypes (data or view) as those are what I can access via sd2df(). With that method, you can then iterate over the results and create a dataframe for every one, or apply some logic to choose some and not others, or whatever; it's python :)

Here's an example and the results from doing this.
Let me know what you think and how you were thinking you'd get dataframes back for each member if other than this. Maybe a dict with names as keys and values being the dataframes?

>>> df = sas.list_tables('sashelp')
>>> df.head()
  MEMNAME MEMTYPE
0  AACOMP    DATA
1   AARFM    DATA
2  ADSMSG    DATA
3   AFMSG    DATA
4     AIR    DATA
>>> len(df)
279
>>> df.tail()
      MEMNAME MEMTYPE
274   ZIPCODE    DATA
275    ZIPMIL    DATA
276        ZT    DATA
277       ZTC    DATA
278  _CMPIDX_    DATA
>>> df[260:266]
     MEMNAME MEMTYPE
260   VTITLE    VIEW
261    VVIEW    VIEW
262   VXATTR    VIEW
263  WAEXPRT    DATA
264   WAPRCS    DATA
265  WATOOLS    DATA
>>>
>>> for row in df.itertuples(index=False):
...    if row[1] == 'VIEW':
...       print('Skipping member '+row[0]+' as it is a VIEW')
...    else:
...       if row[0] in ['AIRLINE', 'BASEBALL', 'CARS']:
...          globals()[row[0]] = sas.sd2df(row[0], 'sashelp')
...       else:
...          print('Skipping member '+row[0]+' of type DATA')
...
Skipping member AACOMP of type DATA
Skipping member AARFM of type DATA
Skipping member ADSMSG of type DATA
Skipping member AFMSG of type DATA
Skipping member AIR of type DATA
Skipping member APPLIANC of type DATA
[...]  removing a bunch to shorten this
Skipping member VREFCON as it is a VIEW
Skipping member VREMEMB as it is a VIEW
Skipping member VSACCES as it is a VIEW
Skipping member VSCATLG as it is a VIEW
[...]  removing a bunch to shorten this
Skipping member ZIPCODE of type DATA
Skipping member ZIPMIL of type DATA
Skipping member ZT of type DATA
Skipping member ZTC of type DATA
Skipping member _CMPIDX_ of type DATA
>>>
>>>
>>> AIRLINE.head()
        DATE  AIR Region
0 1949-01-01  112    ALL
1 1949-02-01  118    ALL
2 1949-03-01  132    ALL
3 1949-04-01  129    ALL
4 1949-05-01  121    ALL
>>> BASEBALL.head()
                Name       Team  nAtBat  nHits  nHome  nRuns  nRBI  nBB  \
0     Allanson, Andy  Cleveland     293     66      1     30    29   14
1        Ashby, Alan    Houston     315     81      7     24    38   39
2        Davis, Alan    Seattle     479    130     18     66    72   76
3      Dawson, Andre   Montreal     496    141     20     65    78   37
4  Galarraga, Andres   Montreal     321     87     10     39    42   30

   YrMajor  CrAtBat    ...      CrBB    League  Division  Position  nOuts  \
0        1      293    ...        14  American      East         C    446
1       14     3449    ...       375  National      West         C    632
2        3     1624    ...       263  American      West        1B    880
3       11     5628    ...       354  National      East        RF    200
4        2      396    ...        33  National      East        1B    805

  nAssts nError Salary  Div  logSalary
0     33     20    NaN   AE        NaN
1     43     10  475.0   NW   6.163315
2     82     14  480.0   AW   6.173786
3     11      3  500.0   NE   6.214608
4     40      4   91.5   NE   4.516339

[5 rows x 24 columns]
>>> CARS.head()
    Make           Model   Type Origin DriveTrain   MSRP  Invoice  EngineSize  \
0  Acura             MDX    SUV   Asia        All  36945    33337         3.5
1  Acura  RSX Type S 2dr  Sedan   Asia      Front  23820    21761         2.0
2  Acura         TSX 4dr  Sedan   Asia      Front  26990    24647         2.4
3  Acura          TL 4dr  Sedan   Asia      Front  33195    30299         3.2
4  Acura      3.5 RL 4dr  Sedan   Asia      Front  43755    39014         3.5

   Cylinders  Horsepower  MPG_City  MPG_Highway  Weight  Wheelbase  Length
0        6.0         265        17           23    4451        106     189
1        4.0         200        24           31    2778        101     172
2        4.0         200        22           29    3230        105     183
3        6.0         270        20           28    3575        108     186
4        6.0         225        18           24    3880        115     197
>>>

Blindly creating object with names of whatever member names show up probably isn't a good idea.
Could do something to scope them better, like maybe

name = 'sashelp_'+row[0].lower()+'_'+row[1]
globals()[name] = sas.sd2df(row[0], 'sashelp')

That would get you the following dataframe objects:

>>> sashelp_
sashelp_airline_DATA   sashelp_baseball_DATA  sashelp_cars_DATA

>>> sashelp_airline_DATA.head()
        DATE  AIR Region
0 1949-01-01  112    ALL
1 1949-02-01  118    ALL
2 1949-03-01  132    ALL
3 1949-04-01  129    ALL
4 1949-05-01  121    ALL
>>>

What do you think?
Tom

[mailbagrahul]

Awesome. This should resolve my use case. I will implement this and let you know how it goes. Thanks a lot.

Do you have any sample proj/code using class&object to interact with sas?

[tomweber-sas]

Cool. I've pushed this to master. Can you grab it from there and try it out? Is that what you were asking about 'sample proj/code using class&object to interact with sas'; where the method?
The only other example is above. This method gives you the list of members and from there, you can implement any logic you want to interact with them.
Let me know how it behaves for you.
Tom

[tomweber-sas]

@mailbagrahul In working on the other iisue 187, adding file_info() method, whch kinda goes with dirlist(), I implemented it like this file_info(), where it returned a dataframe (similar implementation). But, for that case a dictionary really seemed to make more sense, and I like saspy to work even w/out Pandas.
So, I figured a way to return a dict for that method. Looking at list_tables(), I think it might be simpler to return a list of tuples (memname,memtype). It's a lot simpler to iterate over a list of memname,memtype tuples.
Here's the same example but with this:

>>> lf = sas.list_tables_list('sashelp')
>>> lf
[('AACOMP', 'DATA'), ('AARFM', 'DATA'), ('ADSMSG', 'DATA'), ('AFMSG', 'DATA'), 
('AIR', 'DATA'), ('AIRLINE', 'DATA'), ('APPLIANC', 'DATA'), ('ASSCMGR', 'DATA'), 
('ASSOCWA', 'DATA'), ('AUTHLIB', 'DATA'), ('BASEBALL', 'DATA'), ('BDVITEM', 'DATA'), 
('BDVLD3', 'DATA'), ('BDVMETH', 'DATA'), ('BEI', 'DATA'), ('BMIMEN', 'DATA'), 
[...]  removing a bunch to shorten this
('VGOPT', 'VIEW'), ('VIDMSG', 'DATA'), ('VINDEX', 'VIEW'), ('VINFOMP', 'VIEW'), 
('VLIBNAM', 'VIEW'), ('VLOCALE', 'VIEW'), ('VMACRO', 'VIEW'), ('VMEMBER', 'VIEW'), 
('VOPTION', 'VIEW'), ('VOTE1980', 'DATA'),
 [...]  removing a bunch to shorten this
('ZIPCODE', 'DATA'), ('ZIPMIL', 'DATA'), ('ZT', 'DATA'), ('ZTC', 'DATA'), ('_CMPIDX_', 'DATA')]
>>>
>>> for item in lf:
...    print(item[0]+'.'+item[1])
...
AACOMP.DATA
AARFM.DATA
ADSMSG.DATA
AFMSG.DATA
AIR.DATA
AIRLINE.DATA
APPLIANC.DATA
ASSCMGR.DATA
[...]  removing a bunch to shorten this
VCFORMAT.VIEW
VCHKCON.VIEW
VCNCOLU.VIEW
VCNTABU.VIEW
VCOLUMN.VIEW
VDATAIT.VIEW
[...]  removing a bunch to shorten this
ZIPCODE.DATA
ZIPMIL.DATA
ZT.DATA
ZTC.DATA
_CMPIDX_.DATA
>>> lf[:5]
[('AACOMP', 'DATA'), ('AARFM', 'DATA'), ('ADSMSG', 'DATA'), ('AFMSG', 'DATA'), ('AIR', 'DATA')]
>>>
>>> for row in lf:
...     if row[1] == 'VIEW':
...        print('Skipping member '+row[0]+' as it is a VIEW')
...     else:
...        if row[0] in ['AIRLINE', 'BASEBALL', 'CARS']:
...           globals()[row[0]] = sas.sd2df(row[0], 'sashelp')
...        else:
...           print('Skipping member '+row[0]+' of type DATA')
...
Skipping member AACOMP of type DATA
Skipping member AARFM of type DATA
Skipping member ADSMSG of type DATA
Skipping member AFMSG of type DATA
[...]  removing a bunch to shorten this

Skipping member ZH of type DATA
Skipping member ZHC of type DATA
Skipping member ZIPCODE of type DATA
Skipping member ZIPMIL of type DATA
Skipping member ZT of type DATA
Skipping member ZTC of type DATA
Skipping member _CMPIDX_ of type DATA
>>>
>>>
>>>
>>> AIRLINE.head()
        DATE  AIR Region
0 1949-01-01  112    ALL
1 1949-02-01  118    ALL
2 1949-03-01  132    ALL
3 1949-04-01  129    ALL
4 1949-05-01  121    ALL
>>> BASEBALL.head()
                Name       Team  nAtBat  nHits  nHome  nRuns  nRBI  nBB  \
0     Allanson, Andy  Cleveland     293     66      1     30    29   14
1        Ashby, Alan    Houston     315     81      7     24    38   39
2        Davis, Alan    Seattle     479    130     18     66    72   76
3      Dawson, Andre   Montreal     496    141     20     65    78   37
4  Galarraga, Andres   Montreal     321     87     10     39    42   30

   YrMajor  CrAtBat    ...      CrBB    League  Division  Position  nOuts  \
0        1      293    ...        14  American      East         C    446
1       14     3449    ...       375  National      West         C    632
2        3     1624    ...       263  American      West        1B    880
3       11     5628    ...       354  National      East        RF    200
4        2      396    ...        33  National      East        1B    805

  nAssts nError Salary  Div  logSalary
0     33     20    NaN   AE        NaN
1     43     10  475.0   NW   6.163315
2     82     14  480.0   AW   6.173786
3     11      3  500.0   NE   6.214608
4     40      4   91.5   NE   4.516339

[5 rows x 24 columns]
>>>

So, my question is would you like a simple list instead of it being a dataframe?
I think I'm going to make both functions take a option for which result type you want, with the list and dict being the defaults, and then you can ask for a dataframe if you want instead,

What do you think?
Thanks,
Tom

[tomweber-sas]

Ok, I pushed to master the new version, Now you get a list, but if you specify results-;pandas' you get the dataframe:

>>> df = sas.list_tables('sashelp', results='pandas')
>>> df
      MEMNAME MEMTYPE
0      AACOMP    DATA
1       AARFM    DATA
2      ADSMSG    DATA
3       AFMSG    DATA
[...]

>>> lf = sas.list_tables('sashelp')
>>> lf
[('AACOMP', 'DATA'), ('AARFM', 'DATA'), ('ADSMSG', 'DATA'), ('AFMSG', 'DATA'),
[...]

I did the same w/ the file_info() wich is also now at master, for #187.

Tom

[mailbagrahul]

Cool. List should be lot of simpler for processing.

Thanks for doing this to us.

[mailbagrahul]

Can we able to find the username using the process id? Not in saspy. Just a general SAS question.

I'm getting below error. I want to find out who is holding the dataset.

ERROR: A lock is not available for REPORT._RUN_STAT.DATA
ERROR: Lock held by process 26937.

Any thoughts?

[tomweber-sas]

Well, where's SAS? Windows or Linux? local or remote? Can you look at pids on the system it's running on?
Submit youe SASsession object and see what the SASpid is; it could be open in your own session maybe.
see SAS process Pid value = as below.

>>> import saspy
>>> sas = saspy.SASsession(cfgname='sdssas')
SAS Connection established. Subprocess id is 3912

>>> sas
Access Method         = STDIO
SAS Config name       = sdssas
WORK Path             = /sastmp/SAS_workC8C400000F64_tom64-3/
SAS Version           = 9.04.01M4D11092016
SASPy Version         = 2.4.1
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = wlatin1
Python Encoding value = cp1252
SAS process Pid value = 3940

[mailbagrahul]

I think it is Linux and remote. We are connecting thru citrix. I want see in SAS eg not saspy.

[tomweber-sas]

Can you use the 'x' command in that scenario?
In SAS code, you can submit OS commanfs with the X command, if it's not disabled which it can be.
But if not, use
ps -ef linux command.

x ls -la;
  1? x ls -la;
total 732
drwxr-xr-x 6 sastpw r&d  4096 Dec  5 16:01 .
drwxr-xr-x 7 sastpw r&d  4096 Nov 30 10:53 ..
-rwxr-xr-x 1 sastpw r&d  2874 Nov 30 10:53 autocfg.py
drwxr-xr-x 4 sastpw r&d  4096 Nov 30 10:53 doc
-rw-r--r-- 1 sastpw r&d  1469 Nov 30 10:53 __init__.py
drwxr-xr-x 3 sastpw r&d  4096 Nov 30 10:53 java
-rw-r--r-- 1 sastpw r&d  3662 Nov 30 10:53 libname_gen.sas
drwxr-xr-x 2 sastpw r&d  4096 Dec  5 16:02 __pycache__
-rw-r--r-- 1 sastpw r&d 59270 Dec  5 16:01 sasbase.py
[...]  removing a bunch to shorten this
-rw-r--r-- 1 sastpw r&d 73049 Jan 22  2018 titanic.csv
-rw-r--r-- 1 sastpw r&d    22 Dec  4 16:52 version.py
-rw-r--r-- 1 sastpw r&d    22 Nov  7 09:32 version.py.bak
  2? x ps -ef;
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Nov26 ?        00:00:00 /sbin/init
root         2     0  0 Nov26 ?        00:00:00 [kthreadd]
root         3     2  0 Nov26 ?        00:00:00 [migration/0]
[...]

[mailbagrahul]

Dang. I do not have access to this window. Any sas eg statements?

[tomweber-sas]

I don't know anything specific about EG. Just SAS code you could submit through it. I don't know of OS type SAS commands to interrogate processes. The X command is the one way to submit actual OS commands to do that kind of thing.

[tomweber-sas]

Hey @mailbagrahul , I have an initial upload implementation. See #187 . It still needs some tweaks, but it's a working prototype. Download will come next. But, as I'm out next week, it may not be till the beginning of the year.

[tomweber-sas]

@mailbagrahul I have an initial implementation of download pushed to the upload-download branch. Still some work to do on it but give it a go if you want and see what you think. I've round tripped the same text, html document, and binary executable with upload/download, even cross platform, and it's working. Again, won't be as fast as scp or ftp, but you probably can't use those to access the remote machine, so working, but a little slower should be better than failing immediately :)

Also, as this work is being tracked on #187, can we close this issue, as it was for getting file system info, and that's done?

Thanks!
Tom

[mailbagrahul]

Yes sure. We can close this ticket. I have some question on file directory list. Let me open a new ticket and follow thru on that.
Thanks!