Cross OS client.deploy() fails due to OS specific path issues

[geordielad]

If you deploy from a Windows tabpy client to a Linux tabpy server the deploy will fail. This seems due to the fact that the client constructs part of the destination path on the client platform then concatenates that path part to a server path resulting in a path of the form /bla/bla\bla\bla this is not a valid Linux path.

Here is an example:

`
def ztest(p1, p2, n1, n2):
import math
def zstat(p1, p2, n1, n2):
p1 = float(p1)
try:
return (p1 - p2) / math.sqrt((p1 * (1 - p1))/n1 + (p2 * (1 - p2))/n2)
except TypeError: # handle the first value where p2, n2 return as None
return 0
return [zstat(p1, p2, n1, n2) for (p1, p2, n1, n2) in zip(p1, p2, n1, n2)]

client = tabpy_client.Client('http://tabpyserver:9004/')
client.deploy('ztest', ztest, 'ztest for tabPy Demo')
`
The error is:

WARNING:tabpy_client.query_object:Overwriting existing file '/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tabpy_server/staging\endpoints\ztest\1' when saving query object ERROR:tabpy_client.rest:Error with server response. code=500; text={"info": "Error when /add_endpoint: Traceback\n Traceback (most recent call last):\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tabpy_server/tabpy.py\", line 307, in post\n err_msg = yield self._add_or_update_endpoint('add', name, 1, request_data)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tornado/gen.py\", line 1008, in run\n value = future.result()\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tornado/concurrent.py\", line 232, in result\n raise_exc_info(self._exc_info)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tornado/gen.py\", line 1014, in run\n yielded = self.gen.throw(*exc_info)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tabpy_server/tabpy.py\", line 185, in _add_or_update_endpoint\n yield self._copy_po_future(src_path, target_path)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tornado/gen.py\", line 1008, in run\n value = future.result()\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tornado/concurrent.py\", line 232, in result\n raise_exc_info(self._exc_info)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tornado/gen.py\", line 1014, in run\n yielded = self.gen.throw(*exc_info)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tabpy_server/tabpy.py\", line 231, in _copy_po_future\n ret = yield future\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tornado/gen.py\", line 1008, in run\n value = future.result()\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/concurrent/futures/_base.py\", line 400, in result\n return self.__get_result()\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/concurrent/futures/_base.py\", line 359, in __get_result\n reraise(self._exception, self._traceback)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/concurrent/futures/_compat.py\", line 107, in reraise\n exec('raise exc_type, exc_value, traceback', {}, locals_)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/concurrent/futures/thread.py\", line 61, in run\n result = self.fn(*self.args, **self.kwargs)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tabpy_server/tabpy.py\", line 59, in copy_from_local\n shutil.copytree(localpath, remotepath)\n File \"/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/shutil.py\", line 171, in copytree\n names = os.listdir(src)\nOSError: [Errno 2] No such file or directory: '/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tabpy_server/staging\\\\endpoints\\\\ztest\\\\1'\n\nError type : OSError\nError message : [Errno 2] No such file or directory: '/home/ubuntu/anaconda/envs/Tableau-Python-Server/lib/python2.7/site-packages/tabpy_server/staging\\\\endpoints\\\\ztest\\\\1'\n", "message": "error adding endpoint"}

[Proteusiq]

I experienced this. Deploying a test function from Windows 7 to TabPy in Linux server gives:

ResponseError: (500) error adding endpoint "OSError : [Errno 2] No such file or directory: '/usr/local/lib/python2.7/site-packages/tabpy_server/staging\\endpoints\\exp\\1'"

Linux syntax expect /endpoint/Functionmade/VersionNo but it is receiving Windows syntax \endpoint.

[ghost]

Looks to be the same underlying issue as #35. TabPy client can currently only deploy when it is being run on the same machine is TabPy Server - You cannot publish from one client machine to a different server machine.

[jstrauss18]

Has any progress been made on this issue? I ran into this error today trying to deploy from my local machine to tabpy_server.

tabpy_client.rest.ResponseError: (500) error adding endpoint "FileNotFoundError
: [WinError 3] The system cannot find the path specified: 'C:\\Program Files\
\Python35S\\Lib\\site-packages\\tabpy_server\\staging\\endpoints\\a
dd2\\1'"

[ghost]

@JeffStrauss18, If you are trying to deploy from one machine to another, that currently will not work with the existing code.

[joshua-barber]

@SlewT Any idea of whether that kind of remote deployment is planned in future? And/or any suggestions on a multi-user workflow without everyone doing all development on the same machine?

[jstrauss18]

I have a thought how to do this as a workaround. But I haven't fully designed / implemented it yet.

1. Data scientists will save their Python predictive algorithms onto a share drive that resides locally on the host that TabPy runs.
2. A batch utility will be created that has the tabpy_client.connect, pulls in the business algorithms that has been recently updated on the file share, and has the tapy_client.deploy. The utility will run perhaps every 5 minutes.

[ghost]

@joshua-barber, I'm not sure if Tableau has it on their roadmap. I haven't seen much about what their plans are. I'm sure someone will release a version (or a patch) if Tableau doesn't fairly soon - it's needed. I plan on having a working version before too long. Primarily just finishing up an authentication piece I want in place for it and potentially the ability to make sure individuals don't overwrite other's endpoints.

We don't yet have this implemented in our environment as we will need the remote publishing, ability to login, and a couple of other features before it will be set up in anything other than a test environment. If we would implement a workflow in the current state, it would probably be something like what @JeffStrauss18 mentioned except that we would likely have a couple of individuals who did the publishing after reviewing the code and testing it in a non-production server environment before it went to the prod platform.

[nmannheimer]

Hello All,
This is a known issue that has been added to our backlog for TabPy. While we are currently working on adding secured connectivity to TabPy, we are aware of the need for this request as well.

[jstrauss18]

In reference to my comment from way back on Feb 21, it's now implemented.

[mirekphd]

I still run into this issue in 2020, with TabPy client saving the deployed function into its own container's local path and TabPy server attempting to find the function it is own container (which results in error 500).

The only solution indeed seems to be - as suggested by @jstrauss18 to expose the shared location (in our case: persistent volume claim) that both client and server containers can see. Since the volume will be in the same Openshift namespace, the functions and other pickles will be exposed only to a single user running both client and server containers in the same namespace (so no security problems here).

Edit: I can confirm that it worked (client-server file exchange via shared storage), but tabpy was unable to create a folder in which it tries to initialize its .ini file... surely not ensuring the folder exists before creating the .ini file is a bit of a bug. So the .ini file had to be manually created first in the shared location and then deploying functions to "remote" tabpy server from jupyter notebook started to work.