upload and download methods fail when using stdio connection to SAS running within Docker. #450
Comments
|
Hey @royalsouvenir, I will look into this. I haven't used this APro Container myself, but I understand what you're describing and given that it works for you if you can set both the host and ports, then I expect I can make it work with the right enhancements. The 'localhost' parm was one I added not too long ago, when working on my home network where DNS was returning some arbitrary internet addr instead of my local IP's on my network. That was the only case I had seen there be an issue. So, this seems like case number 2 for this kind of thing. Thus it isn't used everywhere and I only had it for the cases I found. This sounds like APro container is similar but not completely the same. |
|
Thanks @tomweber-sas . Reach out if you need assistance with test cases etc as I'd be happy to contribute. If you are following the example docs for SASpy in the Apro documentation site note that it is slightly incorrect on windows. The Docker IP address doesn't resolve on Windows. You need to use At the moment I have a workaround in place. Basically I'm sub-classing fh = ""
try:
with open(local_file) as f:
fh = f.read()
except (FileNotFoundError, OSError) as e:
raise Exception(e)
logger.debug('fl=%s...', fh[0:76])
code = f"%let _local_file=%nrstr({fh});\n"
code += "data _null_;\n"
code += f' file "{remote_file}";\n'
code += ' put "&_local_file";\n'
code += "run;\n" |
|
Hey, thanks. Yesterday I was able to get Apro container up and running on one of my linux systems. Are you running this on a windows box, or from linux? Now I need to do whatever needs to be done to get ssh enabled. And then try to get connected w/ SASPy and start looking at the actual problem. I do see the section of doc for SASPy (which they spelled wrong :( ), FWIW, I did the Quick Start in the following doc to get to where I am: https://go.documentation.sas.com/doc/en/anprocdc/v_009/dplyviya0ctr/p19vjw0smdpog4n1qsrq3q98siqh.htm |
|
Well, I'd have never figured out that sssd stuff; no instructions at all, but someone gave me what I needed to get that to finally work. So, then I was able to get SASPy to work 3 different ways. I never used 'localhost' as the host, which then makes upload and download work as expected. The 3rd case was from the same machine as where the container was running (which worked as above), but I used the IP addr of the container itself, and port 22 itself (default), based upon the IP addr I got back from running the docker command from the doc in the SASPy section (bullet item 4): which happened to be 172.17.0.2 (no telling what you get). but then my config was: In all 3 cases, up/download worked fin, as did everything else I tried too! Can you try adjusting your config and see if it works as expected? Actually, in bullet 5 is says to use that IP from bullet 4, nevee saying to use 'host' : 'localhost'. So using 'host' : '172.17.0.2' is what it says to use which does work - from the same machine, else use the real host name and the forwarded port (2222). Tom |
|
Hi @tomweber-sas . It sounds like this might be a windows quirk likely due to Docker running through hyperV on windows. I'm using Windows 11 with Docker Desktop for development. I used the quickstart from the Apro docs which is how I found that I needed to use I'll put together some unittest scripts today testing the following. I'll also spin up a linux machine to show the differences. I'll get a colleague to try it out on a Mac as well:
Thanks for your help on this. I appreciate it. |
|
Hi @tomweber-sas , The original issue is a red-herring where I thought I needed I confirmed this by changing networks (or more specifically going to my local bar and testing again over some craft beers...) and it works perfectly. It stopped working when I got back home and I had to disable my DNS. Ultimately I don't think there are any coding changes required and it sounds like you have a concrete use case for when localhost is required. I'm guessing it is probably worth a documentation update for using containers on Windows? I'm still going to spin up a linux and osx vm to test further as I use SASPy for app development and it's good to know the quirks between platforms. Do you need a PR to update docs or test cases? If so, let me know and I'll generate some. So where I've landed: Using the following SAS_config_names = ['apro']
apro = {
'saspath' : '/opt/sas/viya/home/SASFoundation/bin/sas_u8',
'ssh' : 'ssh',
'host' : 'localhost', # Windows... love it, hate it.
'luser' : 'sasdemo',
'port' : 8222,
'encoding': 'utf_8',
'localhost': 'host.docker.internal' #Windows again!
}Testcase: import unittest
import socket
import tempfile
import saspy
class TestAproCloudNative(unittest.TestCase):
"""
Test Requirements:
- A valid Docker container instance of Apro with SSH enabled on the chosen account being used.
- Configuration in sascfg_personal.py with the following set if running on windows:
- port set for the exposed ssh port
- localhost set to ``host.docker.internal``
- The host set to ``localhost`` and not the ip address of the container.
- To see failures set the sascfg_personal.py config to something vanilla like removing the ``localhost`` key or setting ``host`` to the ip address
of the docker container.
"""
@classmethod
def setUpClass(cls) -> None:
# If I wasn't so lazy I'd use the docker SDK here to create a container instance. But alas...
cls.connectionid='gitissuev4'
cls.cfgfile='tests/sascfg_personal.py'
data = '1'
cls.tmpfile = tempfile.NamedTemporaryFile(delete=False)
cls.tmpfile.write(data.encode())
cls.tmpfile.seek(0)
cls.infile = cls.tmpfile.name
super().setUpClass()
@classmethod
def tearDownClass(cls) -> None:
cls.tmpfile.close()
super().tearDownClass()
def test_connection(self):
with saspy.SASsession(cfgname=self.connectionid,cfgfile=self.cfgfile) as sas:
self.assertIsNotNone(sas.SASpid)
def test_submit(self):
code = 'NOTE: success'
with saspy.SASsession(cfgname=self.connectionid,cfgfile=self.cfgfile) as sas:
rc = sas.submit(f'%put {code};')
self.assertEqual(sas.SYSERR(),0,rc.get('LOG'))
def test_upload(self):
socket.setdefaulttimeout(300)
with saspy.SASsession(cfgname=self.connectionid,cfgfile=self.cfgfile) as sas:
rc = sas.upload(self.infile,'/data')
self.assertTrue(rc.get('Success'),rc.get('LOG'))
def test_download(self):
socket.setdefaulttimeout(300)
with saspy.SASsession(cfgname=self.connectionid,cfgfile=self.cfgfile) as sas:
rc = sas.download('unittestdownload.txt','/data/.authinfo')
self.assertTrue(rc.get('Success'),rc.get('LOG'))
if __name__ == '__main__':
unittest.main()and invoked (via pytest): pytest test_apro.py -vDocker config (launchapro.ps1)Write-Host "Launch SAS Analytics Pro"
$image="cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-analytics-pro:0.12.28-20220315.1647335918443"
$run_args="-u root " +
"--name sas-analytics-pro " +
"--rm " +
"--detach " +
"--cap-add AUDIT_WRITE " +
"--cap-add SYS_ADMIN " +
"--hostname sas-analytics-pro " +
"--env RUN_MODE=developer " +
"--env SASLICENSEFILE=license.jwt " +
"--env SASLOCKDOWN=0 " +
"--publish 8080:80 " + # Studio
"--publish 8222:22 " + # SSH
"--volume 'my\windows\path\config:/sasinside' " +
"--volume 'my\windows\path\home\camer:/data' "
$cmd="docker run $run_args $image"
$dockercmd="& $cmd"
Invoke-Expression $dockercmdsascfg_personal.pySAS_config_names = ['apro']
apro = {
'saspath' : '/opt/sas/viya/home/SASFoundation/bin/sas_u8',
'ssh' : 'ssh',
'host' : 'localhost', # Windows... love it, hate it.
'luser' : 'sasdemo',
'port' : 8222,
'encoding': 'utf_8',
'localhost': 'host.docker.internal' #Windows again!
}Once again, thanks for your help on this. |
|
Hey, that's a lot of great work and investigation! And, first, I'm glad you've got a config that's working. Knowing it's a local network windows environment helps in the understanding too! That and Docker together, make things more complicated. While I was figuring out what I did yesterday, after getting Apro configured to be able to connect, I had an internal (SAS) consultant get up with me about a problem he was having and it turned out to be very similar to this (what a coincidence). The first problem he had was that Docker screws up you etc/hosts file (at least on windows) replacing the loopback adapter name (127.0.0.1) from 'localhost' to 'host.docker.internal' which breaks using 'localhost', since that no longer resolves. I can't believe Docker does this, but I've seen it over and over. Once we fixed that, he ran into more issues as he was running python in a container, trying to connect to Arpo (it's own container) and similarly, hostnames of the containers didn't resolve between the two so it was back to having to use IP addrs to get anything to work right. All of this makes sense and works right when using machines that have names and ip addrs that DNS has ever heard of. That's why it worked at the pub but not at home! BTW, that's a genius idea for a testing environment :) I am still going to do a little more investigation on my side as to if/how With your configuration that works, can you send me (you can email if you don't want to post here; Id like to see the real info) the LOGs from your upload and download; they each return a dict w/ 'LOG' and 'Success' (boolean) as the keys. That will show me which code path you happen to be going through and maybe provide a little more insight into your specific situation that may be different than mine. More data points is a good thing. Thanks, |
|
Hi Tom, With your colleague's issue get them to try spinning up their Python / Jupyterlab container with apro via docker-compose. You can simply reference the service name as the host in that case. Alternatively, I publish jupyterlab images on dockerhub with SAS Kernel and SASPy already configured at https://hub.docker.com/u/selerity :) You just need to add keys to each container, and change the jupyter username and uid to the same as your apro container user if you want to share home drives and I turn off stricthostkeychecking as well to make my life easier. I'm working through another fun scenario at the moment where I'm spinning up Apro containers from a management interface product I am developing in Django and the Django app is hosted within a container :) I'm about to tackle the next step of using SASPy to validate connectivity etc. Anyway here is the code and output logs. Thanks again :) sascfg_personal.pygitissuev4 = {
'saspath' : '/opt/sas/viya/home/SASFoundation/bin/sas_u8',
'ssh' : 'ssh',
'host' : 'localhost',
'luser' : 'sasdemo',
'port' : 8222,
'localhost': 'host.docker.internal'
}Test Codeimport tempfile
import socket
import pathlib
import filecmp
import pprint
import saspy
socket.setdefaulttimeout(300)
results = {}
with tempfile.NamedTemporaryFile(delete=False) as f:
f.write(b'1')
f.seek(0)
with saspy.SASsession(cfgname='gitissuev4',cfgfile='tests/sascfg_personal.py') as sas:
# Derive the target locations and name from the tempfile
thefile = pathlib.Path(f.name)
cwd = pathlib.Path.cwd()
filename = f'{thefile.stem}{thefile.suffix}'
local_target = cwd.joinpath(filename)
remote_target = f'/data/{filename}'
print(f'tempfile is {f.name}')
print(f'local target is {local_target}')
print(f'remote target is {remote_target}')
results['upload'] = sas.upload(f.name, '/data')
results['download'] = sas.download(local_target,remote_target)
results['File comparison match'] = filecmp.cmp(local_target, f.name)
f.close()
pprint.pprint(results)Log Output |
|
Ok cool, these are what I was wondering about: I also have similar, where the ip in Apro is, obviously, the 172.17.0.2 (or there abouts), and the ip for the host machine is the real IP of it, as opposed to the loopback address or something. What I found, monitoring my network adapters is that nothing is going through the loopback adapter (127.0.0.1), but nothing is going through the real network adapter either (associated w/ that address; eth0). Rather it's going through a 'docker0' network and another virtual one vethnnnn. I believe those are both things set up from docker, so I don't see any traffic actually going over the actual network; all just local to the host machine. That's what I wanted to look into, given it's using the real ip of the host machine (this is a 'remote' connection from machine to machine); but it (docker) knows it's all local. So that's good. So, I think we're good? I've been multitasking too much to be sure; is there anything we need to do at this point? Anything you still need? Thanks! |
|
Hi, |
|
Yes, that sounds like a plan. BTW, who are you working with; the 'apro team'? I'll see about getting the doc updated. I don't need a PR or anything, can do it here. Just to be sure I'm not missing anything, you have I just want to be sure I understand your use of So, what Thanks again! |
|
Hi @tomweber-sas , You are correct about the hosts file. Docker Desktop adds the following (Not sure what it adds on Linux though. There is a beta version of the product for linux now). where x.x.x.xxx is the preferred IPv4 address of the host. The steps I have need in my
I think the documentation on the SASPy side is minimal, maybe just a mention of Thanks again, |
|
Great, thanks for confirming that; docker does put that in the etc/host along with hijacking the loopback adapter from all other software :) Speaking of which, in your config, you are using the IP of Apro as the But again, the only reason you would need to use the So, as for doc changes, I thing the following are what I'm seeing:
That sound correct to you? |
|
Hi Tom, As for everything else, I think you've nailed it. :) Thanks again, |
|
I've gotten in touch w/ the doc write for APRO; already fixed SASPy spelling :) Gave them some info for adding in localhost config option and pointing right to that doc in saspy. I will make mention of docker in that doc too for localhost config option. Thanks again for all the help! |
|
Hi Tom, |
|
That's awesome! I'm gonna send the link to our doc writer! :) maybe we should just reference it from ours! |
Describe the bug
The Analytics Pro Cloud Native product runs within a Docker container. When using the stdio over ssh connection method, the
downloadanduploadmethods of theSASsessionobject fail due to the configuration of the socket filename.Download Method
For the download method, the
filename sockethost should be set tohost.docker.internal. I have tried to use thelocalhostparameter insascfg_personal.pyto set this however withinsaspy/sasiostdio.py, the localhost parameter is not utilised. Thehostipis just usingsocket.gethostname()which will return the hostname of the saspy client.The download method will work if the socket is set to
filename sock socket 'host.docker.internal:{rtunnel}wherertunnelis thertunnelparameter insascfg_personal.py.Upload Method
The upload method has a similar problem however I have no workaround for it. The socket host is set to
:{rtunnel}however I can't get it to work if changing tohost.docker.internal:{rtunnel}and thesocket.connectmethod to either the IP address of the container orlocalhost.To Reproduce
CAP_ADDattributes to allow SSH connections. Add port mappings as well for SSH and socket connections such as-p 22:8222and-p 4444:8444sasdemoby default by creating a .ssh folder under /data , adding relevant key and setting permissions to correct bits.sascfg_personal.pyas follows:python { 'saspath': '/opt/sas/viya/home/SASFoundation/sas', 'ssh': 'ssh', 'host': 'sasdemo@localhost', 'port': {ssh port binding from docker config}, 'tunnel': {local port binding from docker config}, 'rtunnel': {remote port binding from docker config}, 'localhost': 'host.docker.internal', 'options': ["-fullstimer"], 'encoding': 'utf_8' }session.uploadandsession.download. Both should hang or error out.Simplified Workaround code for Download
As
localhostis private my adapter class sets localhost viaExpected behavior
I can't see a valid workaround at the moment for upload. I simply can't get it to work with using localhost or the ip of the container on the client side. I'm also unsure of the intent of the
localhostparameter and whether or not I'm bastardizing it by using it? As a result I'm really not sure what the right resolution here is as to whether the stdio connection class should be altered to use thelocalhostparameter, extend the class further with additional attributes for docker instances or an entirely new connection method for Docker / k8s?Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
NA
The text was updated successfully, but these errors were encountered: