405 Method not allowed connecting to a remote server

[clayms]

Issue Type: Bug

I am getting the following error when connecting to Jupyter running on a remote server.

Failed to connect to remote Jupyter notebook. 
Check that the Jupyter Server URI setting has a valid running server specified. 
http://***.***.***.***:8888/lab 
Error: Invalid response: 405 Method Not Allowed

However, the link given in the Terminal when launching Jupyter Labs, works and I can run python and pyspark code with no issue.

I use the Python VS Code extension to run PySpark jobs on a AWS cluster of EC2 instances running Cloudera CDH with no issue. I simply start the Jupyter server on the Master node and paste the link into the mywork.code-workspace file. For example:

{
    "settings":  {
        "python.dataScience.jupyterServerURI": "http://***.***.***.***:8888/lab?token=xxx..."
     }
}

Now I am trying to do the same thing from the Master node of an EMR cluster, which is still just a cluster of EC2 instances. Again, the exact same Jupyter Labs server on the same port works without any issue from a browser, but I cannot get it to work in the Python extension for VS Code.

Related reported issues:

• Issue 6309 was closed with no resolution.
• Issue 9679 is specific to Jupyter Hub.

Extension version: 2020.2.64397
VS Code version: Code 1.42.1 (c47d83b293181d9be64f27ff093689e8e7aed054, 2020-02-11T14:45:59.656Z)
OS version: Windows_NT x64 10.0.17763

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz (4 x 2496)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: enabled skia_renderer: disabled_off surface_control: disabled_off surface_synchronization: enabled_on video_decode: enabled viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.88GB (6.96GB free)
Process Argv
Screen Reader	no
VM	0%

[IanMatthewHuff]

Hey @clayms . Thanks for reporting this. microsoft/vscode-python#6309 was only closed as we didn't get solid repro information back on that to investigate. Question, could you please get the info from the "Console" section of Help->Toggle Developer Tools when you try to connect. That has our detail output and would be the first think that I would look at to help diagnose this.

[clayms]

Thanks @IanMatthewHuff .

Only Warnings and Errors pasted below.

[Extension Host] Warn Python Extension: 2020-02-25 23:36:33: startHttpServer=false, will not be used. Temporarily turned off

[Extension Host] Error Python Extension: 2020-02-25 23:36:33: Failed to connect to server [t [Error]: Invalid response: 405 Method Not Allowed	at c:\Users\user\.vscode\extensions\ms-python.python-2020.2.64397\out\client\node_modules\@jupyterlab\services.js:3:375991	at async S.createSession (c:\Users\user\.vscode\extensions\ms-python.python-2020.2.64397\out\client\extension.js:1:487953)	at async S.connect (c:\Users\user\.vscode\extensions\ms-python.python-2020.2.64397\out\client\extension.js:1:485234)	at async t.JupyterSessionManager.startNew (c:\Users\user\.vscode\extensions\ms-python.python-2020.2.64397\out\client\extension.js:75:1013334)	at async d.connect (c:\Users\user\.vscode\extensions\ms-python.python-2020.2.64397\out\client\extension.js:75:986497)	at async c:\Users\user\.vscode\extensions\ms-python.python-2020.2.64397\out\client\extension.js:1:816313] {  response: B [Response] {    size: 0,    timeout: 0,    [Symbol(Body internals)]: { body: [PassThrough], disturbed: false, error: null },    [Symbol(Response internals)]: {      url: 'http://***.***.***.***:8889/lab/api/contents/?1582691793239',      status: 405,      statusText: 'Method Not Allowed',      headers: [C [Headers]],      counter: 0    }  }}

Failed to connect to remote Jupyter notebook. Check that the Jupyter Server URI setting has a valid running server specified. http://***.***.***.***:8889/lab Error: Invalid response: 405 Method Not Allowed

[Extension Host] Error Python Extension: 2020-02-25 23:36:33: DataScience Error [Error: Failed to connect to remote Jupyter notebook.
Check that the Jupyter Server URI setting has a valid running server specified.
http://***.***.***.***:8889/lab
Error: Invalid response: 405 Method Not Allowed	at c:\Users\user\.vscode\extensions\ms-python.python-2020.2.64397\out\client\extension.js:1:817741]

[clayms]

@IanMatthewHuff Did you need any additional information?
Thanks

[IanMatthewHuff]

@clayms So the tricky bit here is that I don't really have an easy way to reproduce this myself. I don't have the same AWS setup available to me, and as you noted you were seeing it on your EMR cluster and not your other AWS cluster.

My AWS knowledge is quite low, but I believe that it's going to be some of the EMR security rules from here:
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-man-sec-groups.html
That are preventing the connection. Jupyter Notebooks don't connect in the same way that we do so it is possible for them to make the connection and for us to not be able to. What I hopefully can do to make this debuggable for you is point out exactly what we are requesting that is causing the 405 error.

From the log, the error is coming from when we start up a new session. (If you are interested here is the location):
https://github.com/microsoft/vscode-python/blob/4002b92d8090a7d52043448a69523110f4ed2bc2/src/client/datascience/jupyter/jupyterSession.ts#L420

For our connections to servers and operations on them we use the jupyterlab services. Given the source location and the error it looks like it's the call to the newUntitled function of the content manager here:
https://github.com/jupyterlab/jupyterlab/blob/b58c2c545586cd8ed33bb2e1b40e1823e61879cf/packages/services/src/contents/index.ts#L1098

So it looks like the service connection is trying to do a POST to this url: http://...:8889/lab/api/contents/?1582691793239 and the server is not allowing it. I'm not sure how that's configured though on your servers end.

[jacquespeeters]

Same error here.

I'm successful with Atom-Hydrogen but I've to use cert-tweaks to enable adding extra certificates to Atom's trust store
https://atom.io/packages/cert-tweaks

I was able to solve my 405 error thank to this answer
https://stackoverflow.com/a/60476625/5247364

[clayms]

@jacquespeeters How did you create your self-signed certificate?

I followed these instructions from IBM, then added the certificate to Chrome following the instructions you linked to.

From the EMR master node:

# create key and cert
openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
# combine key and cert
openssl pkcs12 -inkey key.pem -in certificate.pem -export -out certificate.p12

Downloading certificate.p12 to my local computer, and then adding to Chrome: chrome://settings/privacy > Manage certificates > Import > Select and import certificate.p12 > Restarting VS Code.

Same error.

Should I create either the key.pem and certificate.pem on my local then combine into a certificate?
Do I need to use the original .pem key issued when creating the EMR cluster?

Thanks,

[clayms]

Solved it.

The newer versions of Jupyter start what appears to be something like the Terminal based lynx browser.

No matter which Terminal shell I choose, the output is extremely chaotic after I launch Jupyter, in that the 'documents' that the terminal browser is viewing is intermixed with the output of the Jupyter server.

Through all of that noise, I can use the arrow keys to somehow 'navigate' to a point where the following occurs somewhere in the terminal intermixed with the Jupyter output (usually highlighted, but depends on which Terminal program):

cookie: username-***-***-***-***-****=2|1:0|10:***********|27:username-***-***-***-***-****|44:***********************************k1ZmE=|****************************1bef31e Allow? (Y/N/Always/neVer)

I type A and press enter

Sometimes, and sometimes not, I will see the following in the Terminal for a short time:

Data transfer complete  

Then I can press q and get out of whatever terminal browser thing Jupyter launched and just see the normal Jupyter server output. Copy the full url to the Jupyter server, paste it into VS Code Python extension python.dataScience.jupyterServerURI.

Everything seems to be working after that. No certificates or keys needed.