Excercise 2: Deploy as Webservice

[michalmar]

I am getting error when testing the webservice:
Exception in init: "cannot resolve \'labels\' given input columns: [labelsArray];;\\n\'Project [\'labels]\\n+- Relation[labelsArray#0] parquet\\n"\n2020-08-20 20:07:41,062 | root | ...

I am running Databricks runtime: 7.0 (includes Apache Spark 3.0.0, Scala 2.12)

FULL ERROR:
Out[37]: '/bin/bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by /bin/bash)/bin/bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by /bin/bash)
/bin/bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by /bin/bash)
/bin/bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by /bin/bash)
2020-08-20T20:07:04,728932799+00:00 - gunicorn/run
2020-08-20T20:07:04,729873493+00:00 - rsyslog/run
2020-08-20T20:07:04,730695488+00:00 - iot-server/run
bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by bash)
2020-08-20T20:07:04,734858664+00:00 - nginx/run /usr/sbin/nginx: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libcrypto.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)/usr/sbin/nginx: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libssl.so.1.0.0: no version information available (required by /usr/sbin/nginx)/bin/bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by /bin/bash)EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...2020-08-20T20:07:05,074053610+00:00 - iot-server/finish 1 02020-08-20T20:07:05,081571667+00:00 - Exit code 1 is normal. Not restarting iot-server.
Starting gunicorn 19.9.0
Listening at: http://127.0.0.1:31311 (11)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 38
bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by bash)
bash: /azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/libtinfo.so.5: no version information available (required by bash)
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/home/mmlspark/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.microsoft.ml.spark#mmlspark_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-7f0bfc0e-8830-4998-989e-bdbbf44fd02b;1.0
\tconfs: [default]
\tfound com.microsoft.ml.spark#mmlspark_2.11;0.15 in spark-list
\tfound io.spray#spray-json_2.11;1.3.2 in central
\tfound com.microsoft.cntk#cntk;2.4 in central
\tfound org.openpnp#opencv;3.2.0-1 in central
\tfound com.jcraft#jsch;0.1.54 in central
\tfound org.apache.httpcomponents#httpclient;4.5.6 in central
\tfound org.apache.httpcomponents#httpcore;4.4.10 in central
\tfound commons-logging#commons-logging;1.2 in central
\tfound commons-codec#commons-codec;1.10 in central
\tfound com.microsoft.ml.lightgbm#lightgbmlib;2.1.250 in central
:: resolution report :: resolve 968ms :: artifacts dl 11ms
\t:: modules in use:
\tcom.jcraft#jsch;0.1.54 from central in [default]
\tcom.microsoft.cntk#cntk;2.4 from central in [default]
\tcom.microsoft.ml.lightgbm#lightgbmlib;2.1.250 from central in [default]
\tcom.microsoft.ml.spark#mmlspark_2.11;0.15 from spark-list in [default]
\tcommons-codec#commons-codec;1.10 from central in [default]
\tcommons-logging#commons-logging;1.2 from central in [default]
\tio.spray#spray-json_2.11;1.3.2 from central in [default]
\torg.apache.httpcomponents#httpclient;4.5.6 from central in [default]
\torg.apache.httpcomponents#httpcore;4.4.10 from central in [default]
\torg.openpnp#opencv;3.2.0-1 from central in [default]
\t---------------------------------------------------------------------
\t| | modules || artifacts |
\t| conf | number| search|dwnlded|evicted|| number|dwnlded|
\t---------------------------------------------------------------------
\t| default | 10 | 0 | 0 | 0 || 10 | 0 |
\t---------------------------------------------------------------------

:: problems summary ::
:::: ERRORS
\tunknown resolver repo-1

\tunknown resolver repo-1

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
:: retrieving :: org.apache.spark#spark-submit-parent-7f0bfc0e-8830-4998-989e-bdbbf44fd02b
\tconfs: [default]
\t0 artifacts copied, 10 already retrieved (0kB/11ms)
2020-08-20 20:07:09 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Initialized PySpark session.
Initializing logger
2020-08-20 20:07:23,150 | root | INFO | Starting up app insights client
Starting up app insights client
2020-08-20 20:07:23,151 | root | INFO | Starting up request id generator
Starting up request id generator
2020-08-20 20:07:23,151 | root | INFO | Starting up app insight hooks
Starting up app insight hooks
2020-08-20 20:07:23,151 | root | INFO | Invoking user's init function
Invoking user's init function
Exception in init: "cannot resolve 'labels' given input columns: [labelsArray];;
'Project ['labels]
+- Relation[labelsArray#0] parquet
"
2020-08-20 20:07:41,062 | root | INFO | Users's init has completed successfully
Users's init has completed successfully
2020-08-20 20:07:41,065 | root | INFO | Skipping middleware: dbg_model_info as it's not enabled.
Skipping middleware: dbg_model_info as it's not enabled.
2020-08-20 20:07:41,065 | root | INFO | Skipping middleware: dbg_resource_usage as it's not enabled.
Skipping middleware: dbg_resource_usage as it's not enabled.
2020-08-20 20:07:41,067 | root | INFO | Scoring timeout is found from os.environ: 60000 ms
Scoring timeout is found from os.environ: 60000 ms
2020-08-20 20:07:41,082 | root | INFO | Swagger file not present
Swagger file not present
2020-08-20 20:07:41,082 | root | INFO | 404
404
127.0.0.1 - - [20/Aug/2020:20:07:41 +0000] "GET /swagger.json HTTP/1.0" 404 19 "-" "Go-http-client/1.1"
2020-08-20 20:07:45,665 | root | INFO | Swagger file not present
Swagger file not present
2020-08-20 20:07:45,665 | root | INFO | 404
404
127.0.0.1 - - [20/Aug/2020:20:07:45 +0000] "GET /swagger.json HTTP/1.0" 404 19 "-" "Go-http-client/1.1"
127.0.0.1 - - [20/Aug/2020:20:09:33 +0000] "POST /boaform/admin/formLogin HTTP/1.0" 404 232 "http://51.136.121.254:80/admin/login.asp" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0"
2020-08-20 20:11:17,189 | root | INFO | Validation Request Content-Type
Validation Request Content-Type
2020-08-20 20:11:17,190 | root | INFO | \tHost: localhost:5001
\tHost: localhost:5001
2020-08-20 20:11:17,190 | root | INFO | \tX-Real-Ip: 127.0.0.1
\tX-Real-Ip: 127.0.0.1
2020-08-20 20:11:17,190 | root | INFO | \tX-Forwarded-For: 127.0.0.1
\tX-Forwarded-For: 127.0.0.1
2020-08-20 20:11:17,190 | root | INFO | \tX-Forwarded-Proto: http
\tX-Forwarded-Proto: http
2020-08-20 20:11:17,190 | root | INFO | \tConnection: close
\tConnection: close
2020-08-20 20:11:17,190 | root | INFO | \tContent-Length: 395
\tContent-Length: 395
2020-08-20 20:11:17,190 | root | INFO | \tUser-Agent: python-requests/2.22.0
\tUser-Agent: python-requests/2.22.0
2020-08-20 20:11:17,190 | root | INFO | \tAccept: application/json
\tAccept: application/json
2020-08-20 20:11:17,190 | root | INFO | \tAccept-Encoding: gzip, deflate
\tAccept-Encoding: gzip, deflate
2020-08-20 20:11:17,190 | root | INFO | \tContent-Type: application/json
\tContent-Type: application/json
2020-08-20 20:11:17,190 | root | INFO | \tX-Ms-Request-Id: e7732b18-3174-4e54-8acf-9b9c5200d225
\tX-Ms-Request-Id: e7732b18-3174-4e54-8acf-9b9c5200d225
2020-08-20 20:11:17,191 | root | INFO | Scoring Timer is set to 60.0 seconds
Scoring Timer is set to 60.0 seconds
2020-08-20 20:11:17,193 | root | ERROR | Encountered Exception: Traceback (most recent call last):
File "/var/azureml-server/app.py", line 225, in run_scoring
response = invoke_user_with_timer(service_input, request_headers)
File "/var/azureml-server/app.py", line 295, in invoke_user_with_timer
result = user_main.run(**params)
File "/var/azureml-app/main.py", line 21, in run
return_obj = driver_module.run(**arguments)
File "/var/azureml-app/score_sparkml.py", line 31, in run
return json.dumps({"Exception":trainedModel})
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/init.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/encoder.py", line 180, in default
o.class.name)
TypeError: Object of type 'AnalysisException' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/site-packages/flask/app.py", line 1832, in full_dispatch_request
rv = self.dispatch_request()
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/site-packages/flask/app.py", line 1818, in dispatch_request
return self.view_functionsrule.endpoint
File "/var/azureml-server/app.py", line 142, in score_realtime
return run_scoring(service_input, request.headers, request.environ.get('REQUEST_ID', '00000000-0000-0000-0000-000000000000'))
File "/var/azureml-server/app.py", line 238, in run_scoring
raise RunFunctionException(str(exc))
run_function_exception.RunFunctionException

Encountered Exception: Traceback (most recent call last):
File "/var/azureml-server/app.py", line 225, in run_scoring
response = invoke_user_with_timer(service_input, request_headers)
File "/var/azureml-server/app.py", line 295, in invoke_user_with_timer
result = user_main.run(**params)
File "/var/azureml-app/main.py", line 21, in run
return_obj = driver_module.run(**arguments)
File "/var/azureml-app/score_sparkml.py", line 31, in run
return json.dumps({"Exception":trainedModel})
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/init.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/json/encoder.py", line 180, in default
o.class.name)
TypeError: Object of type 'AnalysisException' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/site-packages/flask/app.py", line 1832, in full_dispatch_request
rv = self.dispatch_request()
File "/azureml-envs/azureml_4a5f3c6fb710c7cc1b3312142097af95/lib/python3.6/site-packages/flask/app.py", line 1818, in dispatch_request
return self.view_functionsrule.endpoint
File "/var/azureml-server/app.py", line 142, in score_realtime
return run_scoring(service_input, request.headers, request.environ.get('REQUEST_ID', '00000000-0000-0000-0000-000000000000'))
File "/var/azureml-server/app.py", line 238, in run_scoring
raise RunFunctionException(str(exc))
run_function_exception.RunFunctionException

2020-08-20 20:11:17,193 | root | INFO | 500
500
127.0.0.1 - - [20/Aug/2020:20:11:17 +0000] "POST /score HTTP/1.0" 500 59 "-" "python-requests/2.22.0"
'

[joelhulen]

I was able to successfully run the notebook, deploy the web service, and test the deployed service. Make sure you follow the lab instructions and use Databricks version 6.5, not 7.0. You can edit your cluster and change the Databricks version.

[michalmar]

OK, I understand - so can somebody depict differences / braking changes or better what needs to be done when running on newer version of Databricks Runtime?

[timahenning]

We may have to review this request during our next lab enhancements.