error 403 on new notebook #754

sam-s · 2020-11-11T21:37:41Z

Problem description

When I click on New Notebook, I get the following remote logs

[I 16:26:45.092 NotebookApp] 302 GET /login (100.107.210.216) 1.16ms
[I 16:26:45.107 NotebookApp] 302 GET / (100.107.210.216) 0.37ms
[I 16:26:51.040 NotebookApp] Creating new notebook in 
[I 16:26:51.252 NotebookApp] Kernel started: d84ce358-14de-4252-996c-c5f5d45188ef, name: python3
[W 16:26:51.484 NotebookApp] Couldn't authenticate WebSocket connection
[W 16:26:51.485 NotebookApp] 403 GET /api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels?session_id=6c0afeab-b74e-4ad5-9466-a2422c9f04f6 (100.107.210.216) 2.30ms referer=None

and backtrace

Debugger entered--Lisp error: (websocket-closed #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))
  signal(websocket-closed (#s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t)))
  (if (websocket-openp websocket) nil (signal 'websocket-closed (list frame)))
  websocket-send(#s(websocket :ready-state open :client-data #s(ein:$websocket :ws #1 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ") #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))
  (closure ((opcode . ping) (frame . #s(websocket-frame :opcode ping :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length 99 :completep nil)) (websocket . #s(websocket :ready-state open :client-data #s(ein:$websocket :ws #6 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ")) cl-struct-websocket-frame-tags cl-struct-websocket-tags t) nil (websocket-send websocket (record 'websocket-frame 'pong (progn (or (progn (and (memq ... cl-struct-websocket-frame-tags) t)) (signal 'wrong-type-argument (list 'websocket-frame frame))) (aref frame 2)) nil t)))()
  funcall((closure ((opcode . ping) (frame . #s(websocket-frame :opcode ping :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length 99 :completep nil)) (websocket . #s(websocket :ready-state open :client-data #s(ein:$websocket :ws #7 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ")) cl-struct-websocket-frame-tags cl-struct-websocket-tags t) nil (websocket-send websocket (record 'websocket-frame 'pong (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... frame))) (aref frame 2)) nil t))))
  (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- (nreverse processing-queue))) (while --dolist-tail-- (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--)))))
  (let ((current-frame) (processing-queue) (start-point 0)) (while (setq current-frame (websocket-read-frame (substring text start-point))) (setq processing-queue (cons (websocket-process-frame websocket current-frame) processing-queue)) (setq start-point (+ start-point (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... current-frame))) (aref current-frame 3))))) (if (> (length text) start-point) (progn (progn (or (progn (and (memq ... cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (let* ((v websocket)) (aset v 17 (substring text start-point)))))) (let ((--dolist-tail-- (nreverse processing-queue))) (while --dolist-tail-- (let ((to-process (car --dolist-tail--))) (funcall to-process) (setq --dolist-tail-- (cdr --dolist-tail--))))))
  websocket-process-input-on-open-ws(#s(websocket :ready-state open :client-data #s(ein:$websocket :ws #1 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ") "<!DOCTYPE HTML>\n<html>\n\n<head>\n    <meta charset=\"...")
  (progn (websocket-process-input-on-open-ws websocket (substring text (or start-point 0))))
  (if (eq 'open (progn (or (progn (and (memq (type-of websocket) cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (aref websocket 1))) (progn (websocket-process-input-on-open-ws websocket (substring text (or start-point 0)))))
  (let ((start-point) (text (concat (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... websocket))) (aref websocket 17)) output)) (header-end-pos)) (progn (or (progn (and (memq (type-of websocket) cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (let* ((v websocket)) (aset v 17 nil))) (if (and (eq 'connecting (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... websocket))) (aref websocket 1)))) (progn (if (and (setq header-end-pos (string-match "\15\n\15\n" text)) (setq start-point (+ 4 header-end-pos))) (progn (condition-case err (progn (websocket-verify-response-code text) (websocket-verify-headers websocket text) (websocket-process-headers ... text)) (error (websocket-close websocket) (funcall ... websocket ... err))) (progn (or (progn ...) (signal ... ...)) (let* (...) (aset v 1 ...))) (websocket-try-callback 'websocket-on-open 'on-open websocket)) (progn (or (progn (and ... t)) (signal 'wrong-type-argument (list ... websocket))) (let* ((v websocket)) (aset v 17 text)))))) (if (eq 'open (progn (or (progn (and (memq ... cl-struct-websocket-tags) t)) (signal 'wrong-type-argument (list 'websocket websocket))) (aref websocket 1))) (progn (websocket-process-input-on-open-ws websocket (substring text (or start-point 0))))))
  websocket-outer-filter(#s(websocket :ready-state open :client-data #s(ein:$websocket :ws #1 :kernel #s(ein:$kernel :url-or-port "https://data-science.k8s.region-001.p-use-1.braze...." :path "Untitled.ipynb" :kernelspec nil :events #<ein:events ein:events-1ff1cbe63972> :api-version 6 :session-id "e89e5459-1173-418f-8d45-c69a2eefd7f2" :kernel-id "bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :shell-channel nil :iopub-channel nil :websocket nil :base-url "/api/kernels" :kernel-url "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd" :ws-url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :stdin-activep nil :username "username" :msg-callbacks #<hash-table equal 0/65 0x1ff1cbe670ad> :oinfo-cache #<hash-table equal 0/65 0x1ff1cbe66db3> :after-start-hook nil :after-execute-hook nil) :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-use-1.braze.co..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ") "HTTP/1.1 403 Forbidden\15\nDate: Wed, 11 Nov 2020 21:...")
  (let ((websocket (process-get process :websocket))) (websocket-outer-filter websocket output))
  (closure ((websocket . #s(websocket :ready-state open :client-data #s(ein:$websocket :ws #4 :kernel ... :closed-by-client t) :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>) :negotiated-protocols nil :negotiated-extensions nil :server-p nil :url "wss://data-science.k8s.region-001.p-..." :protocols nil :extensions nil :custom-header-alist nil :conn #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> :server-conn nil :accept-string "wCTPssTm1pug7dZTTudiz17YgJ4=" :inflight-input "\n              exports: \"typ")) (conn . #<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2>) (key . "jbZIs24ROgwuwwQuX4z/4w==") (url-struct . #s(url :type "wss" :user nil :password nil :host "data-science.k8s.region-001.p-use-1...." :portspec 443 :filename "/api/kernels/bef991cb-ed4f-4e0d-8497..." :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t)) (name . "websocket to wss://data-science.k8s....") (custom-header-alist) (nowait) (on-error . #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>)) (on-close . #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>)) (on-message . #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>)) (on-open . #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>)) (extensions) (protocols) (--cl-rest-- :on-open #f(compiled-function (&rest args2) #<bytecode -0x1906404012e5f044>) :on-message #f(compiled-function (&rest args2) #<bytecode -0x1d16c8078ce11c3c>) :on-close #f(compiled-function (ws) #<bytecode -0x1b16d9bfe81d2169>) :on-error #f(compiled-function (ws action err) #<bytecode -0x13da0dc0cc91d084>)) (url . "wss://data-science.k8s.region-001.p-...") cl-struct-websocket-frame-tags cl-struct-websocket-tags t) (process output) (let ((websocket (process-get process :websocket))) (websocket-outer-filter websocket output)))(#<process websocket to wss://data-science.k8s.region-001.p-use-1.braze.com:443/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels?session_id=e89e5459-1173-418f-8d45-c69a2eefd7f2> "HTTP/1.1 403 Forbidden\15\nDate: Wed, 11 Nov 2020 21:...")

Steps to reproduce the problem

had to do

(defun websocket-ensure-connected (websocket)
  nil)

to work around ahyatt/emacs-websocket#75

System info:

("EIN system info"
 :emacs-version
 "GNU Emacs 28.0.50 (build 3, x86_64-apple-darwin19.6.0, NS appkit-1894.60 Version 10.15.7 (Build 19H15))
 of 2020-11-09"
 :emacs-bzr-version "8dc237270f88a6abce4df9a1235b38288792ab71"
 :window-system ns
 :emacs-variant nil
 :os
 (:uname
  "Darwin BZ-C02XR5CGJG5L 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64
"
  :lsb-release nil)
 :notebook "WARNING: Package(s) not found: notebook
"
 :ipython nil
 :image-types
 (imagemagick png gif tiff jpeg xpm xbm pbm)
 :image-types-available
 (imagemagick png gif tiff jpeg xpm xbm pbm)
 :request
 (:backend curl)
 :ein
 (:version "20201028.1857"
	   :source-dir "/Users/sam.steingold/.config/emacs/elpa/ein-20201028.1857/")
 :lib
 ((:name "websocket"
	 :path "~/.config/emacs/elpa/websocket-20200419.2124/websocket.elc"
	 :featurep t
	 :version-var websocket-version
	 :version "1.12")
  (:name "anaphora"
	 :path "~/.config/emacs/elpa/anaphora-20180618.2200/anaphora.elc"
	 :featurep t
	 :version-var nil
	 :version nil)
  (:name "request"
	 :path "~/.config/emacs/elpa/request-20201026.2324/request.elc"
	 :featurep t
	 :version-var request-version
	 :version "0.3.0")
  (:name "deferred"
	 :path "~/.config/emacs/elpa/deferred-20170901.1330/deferred.elc"
	 :featurep t
	 :version-var deferred:version
	 :version "0.5.0")
  (:name "polymode"
	 :path "~/.config/emacs/elpa/polymode-20200606.1106/polymode.elc"
	 :featurep t
	 :version-var nil
	 :version nil)
  (:name "exec-path-from-shell"
	 :path
	 "~/.config/emacs/elpa/exec-path-from-shell-20201105.2236/exec-path-from-shell.elc"
	 :featurep t
	 :version-var nil
	 :version nil)
  (:name "dash"
	 :path "~/.config/emacs/elpa/dash-20200803.1520/dash.elc"
	 :featurep t
	 :version-var nil
	 :version nil)
  (:name "with-editor"
	 :path "~/.config/emacs/elpa/with-editor-20201030.1232/with-editor.elc"
	 :featurep nil
	 :version-var nil
	 :version nil)))

The text was updated successfully, but these errors were encountered:

dickmao · 2020-11-12T04:19:27Z

If

grep 100.107.210.216 ~/.emacs.d/request/curl-cookie-jar | grep username

returns nothing, which is what I expect, then you need to look at the kernel json files in jupyter --runtime-dir on the server machine. You have claimed your server is "nothing special" but I remain unconvinced.

If it does return an entry, and that entry isn't reflected in M-x url-cookie-list, then there's a problem in ein-websocket.el. This seems unlikely, not because ein-websocket.el is particulary well written, but rather because there's not much that could break there.

sam-s · 2020-11-12T14:24:52Z

the relevant line in ~/.config/emacs/request/curl-cookie-jar is

data-science.k8s.region-001.p-use-1.braze.com	FALSE	/	FALSE	0	_xsrf	2|682c5193|ZZZZ|1603214402

moreover, ein:notebooklist-login does not ask me for a password anymore (and shows the list of notebooks).
however, url-cookie-list fails with "No cookies are defined".

the kernel file created in response to the ein request above is kernel-d84ce358-14de-4252-996c-c5f5d45188ef.json:

{
  "shell_port": 55941,
  "iopub_port": 38979,
  "stdin_port": 34231,
  "control_port": 48685,
  "hb_port": 48739,
  "ip": "127.0.0.1",
  "key": "a9b98a0a-dad9102e9e463e610c965357",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

the chrome web interface seems to have no problems interacting with this kernel.

dickmao · 2020-11-12T15:57:36Z

data-science.k8s.region-001.p-use-1.braze.com	FALSE	/	FALSE	0	_xsrf

That's the cross-site forgery token, which is not relevant. I would rather see something like:

#HttpOnly_127.0.0.1	FALSE	/	FALSE	1607728889	username-127-0-0-1-8317

Thus it appears your server does not ask for a user, which would account for tornado complaining with 403, no current user.

however, url-cookie-list fails with "No cookies are defined".

That is perturbing.

the kernel file created in response to the ein request above is

Hmm.. that didn't help me. Maybe you could report back with the nbserver.*.json

dickmao · 2020-11-12T16:01:59Z

An empty url-cookie-list, btw, is almost certainly a showstopper. I would M-x find-library RET url-cookie and see if you have one of the various defcustoms in there set to a suspect value in your .emacs.

sam-s · 2020-11-12T18:46:25Z

#HttpOnly_127.0.0.1	FALSE	/	FALSE	1607728889	username-127-0-0-1-8317

#HttpOnly_data-science.k8s.region-001.p-use-1.braze.com	FALSE	/	FALSE	1607481770	username-data-science-k8s-region-001-p-use-1-braze-com	"2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:XXXXX=|YYYYYY"

Thus it appears your server does not ask for a user, which would account for tornado complaining with 403, no current user.

I have a modicum of control over how the server is run.
Now it is run as

jupyter notebook --port=8080 --ip=0.0.0.0 --allow-root \
     --NotebookApp.iopub_data_rate_limit=1.0e10 \
     --NotebookApp.iopub_msg_rate_limit=1.0e10

How do you want me to run it?

however, url-cookie-list fails with "No cookies are defined".

That is perturbing.
An empty url-cookie-list, btw, is almost certainly a showstopper. I would M-x find-library RET url-cookie and see if you have one of the various defcustoms in there set to a suspect value in your .emacs.

I don't set anything url-cookie-* in my .emacs; moreover, I run emacs -Q so that cannot be relevant.

moreover, when I run

emacs -Q -f package-initialize --eval "(setq debug-on-error t)" --load websocket.elc --eval '(url-cookie-parse-file-netscape (expand-file-name "request/curl-cookie-jar" user-emacs-directory))'

and M-x url-cookie-list includes

data-science.k8s.region-001.p-use-1.braze.com     _xsrf                2|682c5193|ZZZZ|1603214402
data-science.k8s.region-001.p-use-1.braze.com:443 _xsrf                2|682c5193|ZZZZ|1603214402
                                                  username-data-scienc "2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:XXXX=|YYYY"

I still get the same error

Hmm.. that didn't help me. Maybe you could report back with the nbserver.*.json

{
  "base_url": "/",
  "hostname": "0.0.0.0",
  "notebook_dir": "/root/data/data-science/jupyter",
  "password": false,
  "pid": 1,
  "port": 8080,
  "secure": false,
  "sock": "",
  "token": "ZZZZ",
  "url": "http://0.0.0.0:8080/"
}

dickmao · 2020-11-12T20:08:46Z

I know as little about cookies as the next non-web programmer, but artificially parsing an old jar file is mildly suspect. It's conceivable the server insists on new cookies for each new session.

If it does return an entry, and that entry isn't reflected in M-x url-cookie-list, then there's a problem in ein-websocket.el.

So it sounds like this is the case. The jar file has the username entry, but the in-memory url-cookie-list doesn't, and worse is completely empty, which is very suspect. You should C-u C-M-x in

emacs-ipython-notebook/lisp/ein-websocket.el

Line 73 in bf1ca5b

(defun ein:websocket--prepare-cookies (url)

and see why url-cookie-store isn't firing.

sam-s · 2020-11-12T20:32:35Z

I know as little about cookies as the next non-web programmer, but artificially parsing an old jar file is mildly suspect. It's conceivable the server insists on new cookies for each new session.

this is not an old file - its mtime is today.

If it does return an entry, and that entry isn't reflected in M-x url-cookie-list, then there's a problem in ein-websocket.el.

So it sounds like this is the case. The jar file has the username entry, but the in-memory url-cookie-list doesn't, and worse is completely empty, which is very suspect. You should C-u C-M-x in

emacs-ipython-notebook/lisp/ein-websocket.el

Line 73 in bf1ca5b

(defun ein:websocket--prepare-cookies (url)

and see why url-cookie-store isn't firing.

ein:websocket--prepare-cookies does not call url-cookie-store directly.
cookies is computed and has a reasonable value:

(("username-data-science-k8s-region-001-p-use-1-braze..." . "\"2|1:0|10:1604889770|54:username-data-science-k8s-...") ("_xsrf" . "2|682c5193|ZZZZ|160321..."))

ein:websocket-store-cookie is called twice and each time calls url-cookie-store successfully:

======================================================================
1 -> (url-cookie-store "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZZZ=|XXX\"" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: nil
======================================================================
1 -> (url-cookie-store "_xsrf" "2|682c5193|YYY|1603214402" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: nil

dickmao · 2020-11-12T20:50:54Z

It sounds like you are continuing along the path of woe of force parsing (url-cookie-parse-file-netscape (expand-file-name "request/curl-cookie-jar" user-emacs-directory)). I would not do this. I know the cookies are not chronologically old, but every new websocket instance may require a brand new cookie. Who knows? I certainly don't.

Starting from scratch, you said M-x url-cookie-list was empty. I would try to find out why that is.

sam-s · 2020-11-12T21:50:58Z

It sounds like you are continuing along the path of woe of force parsing (url-cookie-parse-file-netscape (expand-file-name "request/curl-cookie-jar" user-emacs-directory)).

no, I am not doing that.

Starting from scratch, you said M-x url-cookie-list was empty. I would try to find out why that is.

emacs -Q -f package-initialize --eval "(setq debug-on-error t)" --eval '(load "websocket")' --eval '(defun websocket-ensure-connected (websocket) nil)'
M-x ein:notebooklist-login RET https://data-science.k8s.region-001.p-use-1.braze.com/ RET
M-x url-cookie-list RET
==> (error "No cookies are defined")
[New Notebook]
==> (websocket-closed #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))
M-x url-cookie-list RET
==> 
data-science.k8s.region-001.p-use-1.braze.com:443 _xsrf                2|682c5193|ZZZ|1603214402
                                                  username-data-scienc "2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:XXX=|YYY"

sam-s · 2020-11-12T21:56:06Z

emacs -Q -f package-initialize --eval "(setq debug-on-error t)" --eval '(load "websocket")' --eval '(defun websocket-ensure-connected (websocket) nil)'
M-x trace-function RET url-cookie-store RET
M-x ein:notebooklist-login RET https://data-science.k8s.region-001.p-use-1.braze.com/ RET

no *trace-output*.

[New Notebook]
==> (websocket-closed #s(websocket-frame :opcode pong :payload "2e9fb87e0d8ef1ea96893aa6906147a371bb\" type=\"text/c..." :length nil :completep t))

and now *trace-output* is

======================================================================
1 -> (url-cookie-store "_xsrf" "2|682c5193|XXX|1603214402" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: ((#1="data-science.k8s.region-001.p-use-1.braze.com:443" [url-cookie "_xsrf" "2|682c5193|XXX|1603214402" nil "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" #1# 0]))
======================================================================
1 -> (url-cookie-store "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:YYY=|ZZZ\"" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" 0)
1 <- url-cookie-store: ([url-cookie "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1604889770|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:YYY=|ZZZ\"" nil "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" #1="data-science.k8s.region-001.p-use-1.braze.com:443" 0] [url-cookie "_xsrf" "2|682c5193|XXX|1603214402" nil "/api/kernels/bef991cb-ed4f-4e0d-8497-0d5efb3915fd/channels" #1# 0])

dickmao · 2020-11-12T23:07:02Z

Hum, that output looks good.

Your token is non-empty "ZZZZ". Your server should not just let you in without asking for it.

I've seen inconsistent behaviors when invoking the server with --ip=0.0.0.0. If you omit that switch will the server

require the ZZZZ token, and
stop complaining about 403

?

I am approaching the limits of what I can advise remotely. On-site troubleshooting is possible for a nominal fee.

sam-s · 2020-11-12T23:22:19Z

Your token is non-empty "ZZZZ". Your server should not just let you in without asking for it.

this is the token in nbserver-1.json, it was asked the 1st time I used ein to connect to this server, then it was saved in request/curl-cookie-jar and never asked again.

I've seen inconsistent behaviors when invoking the server with --ip=0.0.0.0. If you omit that switch will the server

require the ZZZZ token, and

stop complaining about 403

I have been using --ip=0.0.0.0 since times immemorial; never had a problem.
moreover, I do need to supply something there, otherwise only local connections would be allowed and I am very much remote there...

I can force being asked for the token by removing curl-cookie-jar...

I am approaching the limits of what I can advise remotely. On-site troubleshooting is possible for a nominal fee.

The fee is probably not a problem, access is.

dickmao · 2020-11-12T23:27:59Z

I would be surprised if deleting the cookie jar suddenly caused the server to require ZZZZ. My understanding was the "token" has nothing to do with the cookie business. Either way, it would be instructive to know if a server requiring ZZZZ is better behaved (doesn't 403).

sam-s · 2020-11-12T23:28:50Z

huh, I renamed cookie jar and repeated #754 (comment)
I was asked for password, I entered 288XXXX, found in nbserver-1.json.
Here is *trace-output*:

======================================================================
1 -> (url-cookie-store "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1605223426|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZmIXXXX=|eb5XXX\"" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" 0)
1 <- url-cookie-store: ((#1="data-science.k8s.region-001.p-use-1.braze.com:443" [url-cookie "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1605223426|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZmIXXX=|eb5XXX\"" nil "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" #1# 0]))
======================================================================
1 -> (url-cookie-store "_xsrf" "2|046f02b3|01ff161c84d0637bae97d257f35e12c4|1605223410" nil "data-science.k8s.region-001.p-use-1.braze.com:443" "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" 0)
1 <- url-cookie-store: ([url-cookie "_xsrf" "2|046f02b3|01ffXXX|1605223410" nil "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" #1="data-science.k8s.region-001.p-use-1.braze.com:443" 0] [url-cookie "username-data-science-k8s-region-001-p-use-1-braze-com" "\"2|1:0|10:1605223426|54:username-data-science-k8s-region-001-p-use-1-braze-com|44:ZmIXXX=|eb5XXX\"" nil "/api/kernels/d84ce358-14de-4252-996c-c5f5d45188ef/channels" #1# 0])

note that the password/token (288XXX.....) is not in the cookie jar.
and, of course

(websocket-closed #s(websocket-frame :opcode pong :payload "2e9XXXX\" type=\"text/c..." :length nil :completep t))

2e9XXX is not in the jar either...

dickmao · 2020-11-12T23:36:23Z

Sorry, I can't help you. It'd be a moderate hassle but replicating the server configuration onto your local laptop and attempting to replicate the problem wholly on one machine might be very instructive.

sam-s · 2020-11-12T23:37:40Z

I would be surprised if deleting the cookie jar suddenly caused the server to require ZZZZ.

That's what did happen!

My understanding was the "token" has nothing to do with the cookie business.

My understanding is that when a server authenticates a user, it sets a cookie on the client which the client is supposed to submit in all future communications in lieu of re-authentication (this is why intercepting cookies is so dangerous).
@ahyatt - could you please confirm?

Either way, it would be instructive to know if a server requiring ZZZZ is better behaved (doesn't 403).

No change.

dickmao · 2020-11-13T00:38:18Z

My understanding was the "token" has nothing to do with the cookie business.

Ah, you're right. The above statement is a complete falsehood.

sam-s · 2020-11-13T16:47:13Z

just to make it clear - the problem is HTTP 403 error when connecting to jupyter server behind a proxy

ahyatt · 2020-11-13T20:59:21Z

Are you familiar with ahyatt/emacs-websocket#73? Right now websockets don't work correctly with the rest of emacs connection proxying. There's a fix, but it's kind of half-baked at the moment. You can try the pull request in the fix and see if it helps you.

About server authentication & cookies - this really depends on what exactly the server is doing. Normal HTTP authentication doesn't use cookies, but does requires the Authorization header to be included on each request. However, I'm sure there are many variants, and I'm not super well informed about them.

You should authenticate via whatever HTTP method your server supports at the beginning of the websocket session. Further communication shouldn't need anything, and it's only needed again when you re-connect. Those re-connections should be rare, since the websockets can last a while.

dickmao closed this as completed Nov 12, 2020

sam-s mentioned this issue Nov 13, 2020

error 403 with ein tkf/emacs-request#193

Closed

dickmao mentioned this issue Nov 14, 2020

Support connecting through proxies ahyatt/emacs-websocket#73

Open

sam-s mentioned this issue Nov 24, 2020

error 403 with ein over proxy ahyatt/emacs-websocket#76

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error 403 on new notebook #754

error 403 on new notebook #754

sam-s commented Nov 11, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020 •

edited

dickmao commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020 •

edited

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 13, 2020

sam-s commented Nov 13, 2020

ahyatt commented Nov 13, 2020

error 403 on new notebook #754

error 403 on new notebook #754

Comments

sam-s commented Nov 11, 2020

Problem description

Steps to reproduce the problem

System info:

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020 • edited

dickmao commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020 • edited

dickmao commented Nov 12, 2020

sam-s commented Nov 12, 2020

dickmao commented Nov 13, 2020

sam-s commented Nov 13, 2020

ahyatt commented Nov 13, 2020

sam-s commented Nov 12, 2020 •

edited

sam-s commented Nov 12, 2020 •

edited