Issue pushing swat CAS Python Model to SAS Model Manager using sasctl

[kichil]

Describe the issue
I created a model in Python in JupyterHub using cas and swat. When attempting to push the model from JupyterHub to SAS Model Manager using sasctl I am receiving the following error: HTTP Error 422. I do get the following success before receiving the error: NOTE: Cloud Analytic Services saved the file _C7E01D4A82C44B3B94567BF95.sashdat in caslib ModelStore. Then I receive the HTTP Error 422.

I am able to register the model via a workaround using sas code similar to the following:
%mm_get_project_id(projectNm=%str(TS_7613611429), projectLoc=%str(Public),
idvar=myProjID);
%mm_import_astore_model(locationID=%str(&myprojID),
modelname=%str(Model2forKirsten), modeldesc=%str(),
rstore=MSTORE._C7E01D4A82C44B3B94567BF95);

To Reproduce
Please see attached file CAS Python HMEQ to SAS MM.

Expected behavior
I expect a new project to be created in Model Manager and my Python model to be loaded into the project without needing to write sas code (register the model directly from jupyterhub).

Stack Trace
---> 10 model = register_model(astore_gb, 'HMEQ_Test_KC', 'Python_HMEQ_KC', force=True)

/opt/anaconda3/lib/python3.8/site-packages/sasctl/tasks.py in register_model(model, name, project, repository, input, version, files, force, record_packages)
314 project = _create_project(project, model_props, repo_obj, invar, outvar)
315

--> 316 model = mr.import_model_from_zip(name, project, zipfile, version=version)
317 return model
318

/opt/anaconda3/lib/python3.8/site-packages/sasctl/_services/model_repository.py in import_model_from_zip(cls, name, project, file, description, version)
525 params = "&".join("{}={}".format(k, v) for k, v in params.items())
526
--> 527 r = cls.post(
528 "/models#octetStream",
529 data=file.read(),

/opt/anaconda3/lib/python3.8/site-packages/sasctl/_services/service.py in post(cls, *args, **kwargs)
111 def post(cls, *args, **kwargs):
112 """Send a POST request."""
--> 113 return cls.request("post", *args, **kwargs)
...
-> 1630 raise HTTPError(
1631 response.url, response.status_code, response.text, response.headers, None
1632 )

HTTPError: HTTP Error 422: {"links":[{"method":"GET","rel":"self","href":"/modelRepository/models?start=0&limit=1","uri":"/modelRepository/models?start=0&limit=1","type":"application/vnd.sas.collection","itemType":"application/vnd.sas.models.model.summary"},{"method":"POST","rel":"createModel","href":"/modelRepository/models","uri":"/modelRepository/models","type":"application/vnd.sas.models.model","responseType":"application/vnd.sas.models.model"}],"name":"models","start":0,"count":1,"items":[{"errorCode":0,"message":"","links":[],"version":2,"httpStatusCode":422}],"limit":1,"version":2}

Version
I am trying to push to a Viya 4 environment running 2022.1.2. I am using the following version of sasctl: 1.7.2

CAS Python HMEQ to SAS MM.zip

[smlindauer]

Hey @kichil,

Thank you for the attached jupyter notebook.

I have replicated this error on my end. This does not look like its an error with a python-sasctl. I have posted a bug internally and will update when a fix has been released to SAS Viya. For your reference, I see the following output when using the session.add_stderr_logger():

HTTP/1.1 POST http://sasdemo.sas.com/modelRepository/models?name=HMEQ_Test&description=None&type=ZIP&projectId=c251f5f6-f457-44cf-b9d5-57e0ac9e3418&versionOption=new#octetStream
User-Agent: python-requests/2.25.1
Accept-Encoding: gzip, deflate
Accept: /
Connection: keep-alive
Content-Type: application/octet-stream
Cookie: sas-ingress-nginx=016585dfb2594fbecbd3c0d1f93bc4e3|1325d69d3da3dda0f2003095cc4fc02e
Content-Length: 13215
Authorization: Bearer [redacted]
Body:
<SNIPPED FOR LENGTH>
HTTP 422 http://sasdemo.sas.com/modelRepository/models?name=HMEQ_Test&description=None&type=ZIP&projectId=c251f5f6-f457-44cf-b9d5-57e0ac9e3418&versionOption=new#octetStream
Date: Mon, 18 Jul 2022 14:09:24 GMT
Content-Type: application/vnd.sas.models.model.spk+json
Transfer-Encoding: chunked
Vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
SAS-Service-Response-Flag: true
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: SAMEORIGIN
Body:
{'count': 1,
'items': [{'errorCode': 0,
'httpStatusCode': 422,
'links': [],
'message': '',
'version': 2}],
'limit': 1,
'links': [{'href': '/modelRepository/models?start=0&limit=1',
'itemType': 'application/vnd.sas.models.model.summary',
'method': 'GET',
'rel': 'self',
'type': 'application/vnd.sas.collection',
'uri': '/modelRepository/models?start=0&limit=1'},
{'href': '/modelRepository/models',
'method': 'POST',
'rel': 'createModel',
'responseType': 'application/vnd.sas.models.model',
'type': 'application/vnd.sas.models.model',
'uri': '/modelRepository/models'}],
'name': 'models',
'start': 0,
'version': 2}

[kichil]

@smlindauer thank you for the update. There is another team that can run similar code in their Viya 3.5 environment without issue. We were wondering if it is related to our use of Viya 4.

[smlindauer]

@kichil I am not entirely sure what has changed with the API endpoint from Viya 3.5 to Viya 4 because the call is the same for both versions (POST /modelRepository/models#octetStream). I'll investigate further today and see what I can find out. The endpoint works for Python zip models, but is stalling with astores, so it is likely an issue with content acceptance somewhere in the API definition.

I'll also see if there are some alternative approaches that might work in the meantime (or possibly as a replacement for the method).

[smlindauer]

I'm not reopening the previous issue, but I do want to note that issue #115 does report the same problem, which I had previously verified to be working in Viya 4 on version 2022.1.2 and later.

[kichil]

@smlindauer Thank you for the comments. I read through the aforementioned issue and it looks like they used a pzmm file to pass to Model Manager. I have other scripts that are able to successfully pass pzmm files to Model Manager. I am attempting to push the model (astore file) without creating pzmm files, to reduce complexity in the process when explaining to clients the process.

Since a colleague was able to successfully run the code in a different environment in 3.5, it appears that the code should work. Maybe it is in the environment set up or as you mentioned the API definition. Thank you for your support!

[kichil]

@smlindauer Checking in, any progress towards a resolution or further insights into the error?

[smlindauer]

@kichil Still waiting on a response internally as to why the /modelRepository/models#octetStream API is not working as expected. I will update when I get a response. In the meantime, let me research to see if there is an alternative API we could try to leverage.

[smlindauer]

@kichil: A fix has been pushed on Viya 4, however the new version has not yet been released. This issue should be fixed as of Viya 4 version 2022.09.

I'm going to hold this open until the release in case we hit any more hiccups.

[kichil]

@smlindauer Thank you for the update! I am glad to hear the issue was identified and resolved. Once the new version is released I will give it a try and let you know how it goes. Thank you.

[smlindauer]

@kichil I refactored the entire astore upload call in the most recent release. This bypasses the API bug we've been seeing where a 422 was being returned. Furthermore, this removes the onus of score code generation from a client side operation (via sasctl) and back to a server side operation (via SAS Model Manager identifying the import as an astore model). Hopefully, this should also minimize any future errors we may see when scoring these models since we'll be dependent upon the code generated by MM and then used by MM.

Updating to the newest release, v1.7.3 on PyPI, should allow your swat CAS Python Model to be successfully uploaded and scored in SAS Model Manager.

[kichil]

@smlindauer All good, works swimmingly. Thank you!