Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FlightRPC][Python] Flight RPC crashes when a middleware sends an authorization header written with an upper-case A as in 'Authorization' #31960

Closed
asfimport opened this issue May 18, 2022 · 6 comments
Closed

[FlightRPC][Python] Flight RPC crashes when a middleware sends an authorization header written with an upper-case A as in 'Authorization' #31960

asfimport opened this issue May 18, 2022 · 6 comments
Assignees
@lidavidm
Labels
Component: FlightRPC Component: Python Type: bug
Milestone
9.0.0

Comments

@asfimport
Copy link
Collaborator

Sending a custom Authorization header leads to a crash of the client

 

Running this python code, for example

 

import pyarrow.flight as flight

class TestMiddlewareFactory(ClientMiddlewareFactory):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def start_call(self, info):
        return TestMiddleware()


class TestMiddleware(ClientMiddleware):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def sending_headers(self):
        return {"Authorization": "Basic dXNlcjpwYXNz"}


def test():
    client = flight.FlightClient("grpc://localhost:8491", middleware=[TestMiddlewareFactory()])
    client.do_get(flight.Ticket(""))

 

 

Results in

 

 


tests/rpc_repro.py Fatal Python error: AbortedCurrent thread 0x0000000202ecc600 (most recent call first):
  File "tests/rpc_repro.py", line 22 in test
  File "venv/lib/python3.9/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "venv/lib/python3.9/site-packages/_pytest/python.py", line 1641 in runtest
  File "venv/lib/python3.9/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "venv/lib/python3.9/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "venv/lib/python3.9/site-packages/_pytest/runner.py", line 311 in from_call
  File "venv/lib/python3.9/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "venv/lib/python3.9/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "venv/lib/python3.9/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "venv/lib/python3.9/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "venv/lib/python3.9/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "venv/lib/python3.9/site-packages/_pytest/main.py", line 323 in _main
  File "venv/lib/python3.9/site-packages/_pytest/main.py", line 269 in wrap_session
  File "venv/lib/python3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162 in main
  File "venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "venv/bin/pytest", line 8 in <module>
Abort trap: 6

 

 

With an additional crash report from the OS

 


Process:               Python [26728]
Path:                  /usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Identifier:            org.python.python
Version:               3.9.12 (3.9.12)
Code Type:             X86-64 (Translated)
Parent Process:        bash [4683]
Responsible:           iTerm2 [99236]
User ID:               501Date/Time:             2022-05-18 15:35:10.1978 +0200
OS Version:            macOS 12.3.1 (21E258)
Report Version:        12
Anonymous UUID:        4A72633D-06AC-F2CE-0E3F-0AD87FA611CESleep/Wake UUID:       3D7BD416-99A9-41B3-8163-5544AEF31FF5Time Awake Since Boot: 1000000 seconds
Time Since Wake:       22827 secondsSystem Integrity Protection: enabledCrashed Thread:        0  Dispatch queue: com.apple.main-threadException Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFYApplication Specific Information:
abort() called
Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   ???                                 0x7ff8a597e940 ???
1   libsystem_kernel.dylib              0x7ff81537300e __pthread_kill + 10
2   libsystem_pthread.dylib             0x7ff8153a91ff pthread_kill + 263
3   libsystem_c.dylib                   0x7ff8152b72c8 raise + 26
4   libsystem_platform.dylib            0x7ff8153bedfd _sigtramp + 29
5   ???                                 0x206e69207961 ???
6   libsystem_c.dylib                   0x7ff8152f4d24 abort + 123
7   libarrow_flight.700.dylib              0x1333dac17 grpc::CoreCodegen::assert_fail(char const*, char const*, int) + 39
8   libarrow_flight.700.dylib              0x13337f4a2 grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpClientSendClose, grpc::internal::CallNoOp<4>, grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::ContinueFillOpsAfterInterception() + 258
9   libarrow_flight.700.dylib              0x13337aa07 arrow::flight::GrpcClientInterceptorAdapter::Intercept(grpc::experimental::InterceptorBatchMethods*) + 295
10  libarrow_flight.700.dylib              0x13337dd76 grpc::internal::InterceptorBatchMethodsImpl::RunInterceptors() + 294
11  libarrow_flight.700.dylib              0x13337f350 grpc::internal::CallOpSet<grpc::internal::CallOpSendInitialMetadata, grpc::internal::CallOpSendMessage, grpc::internal::CallOpClientSendClose, grpc::internal::CallNoOp<4>, grpc::internal::CallNoOp<5>, grpc::internal::CallNoOp<6> >::FillOps(grpc::internal::Call*) + 256
12  libarrow_flight.700.dylib              0x1333affb1 grpc::ClientReader<arrow::flight::protocol::FlightData>::ClientReader<arrow::flight::protocol::Ticket>(grpc::ChannelInterface*, grpc::internal::RpcMethod const&, grpc::ClientContext*, arrow::flight::protocol::Ticket const&) + 689
13  libarrow_flight.700.dylib              0x13335b1cf arrow::flight::FlightClient::FlightClientImpl::DoGet(arrow::flight::FlightCallOptions const&, arrow::flight::Ticket const&, std::__1::unique_ptr<arrow::flight::FlightStreamReader, std::__1::default_delete<arrow::flight::FlightStreamReader> >*) + 255
14  libarrow_flight.700.dylib              0x13335b0c1 arrow::flight::FlightClient::DoGet(arrow::flight::FlightCallOptions const&, arrow::flight::Ticket const&, std::__1::unique_ptr<arrow::flight::FlightStreamReader, std::__1::default_delete<arrow::flight::FlightStreamReader> >*) + 17
15  _flight.cpython-39-darwin.so           0x1328544b7 __pyx_pw_7pyarrow_7_flight_12FlightClient_22do_get(_object*, _object*, _object*) + 231
16  Python                                 0x108f05c5a method_vectorcall_VARARGS_KEYWORDS + 176
17  Python                                 0x108fe7ca4 _PyEval_EvalFrameDefault + 22263
18  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
19  Python                                 0x108fe7ca4 _PyEval_EvalFrameDefault + 22263
20  Python                                 0x108fe142c _PyEval_EvalCode + 398
21  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
22  Python                                 0x108fe7ca4 _PyEval_EvalFrameDefault + 22263
23  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
24  Python                                 0x108efcb53 method_vectorcall + 160
25  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
26  Python                                 0x108fe142c _PyEval_EvalCode + 398
27  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
28  Python                                 0x108efcb53 method_vectorcall + 160
29  Python                                 0x108fe79bc _PyEval_EvalFrameDefault + 21519
30  Python                                 0x108fe142c _PyEval_EvalCode + 398
31  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
32  Python                                 0x108ef9c92 PyVectorcall_Call + 164
33  Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
34  Python                                 0x108fe142c _PyEval_EvalCode + 398
35  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
36  Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
37  Python                                 0x108fe142c _PyEval_EvalCode + 398
38  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
39  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
40  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
41  Python                                 0x108efcb53 method_vectorcall + 160
42  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
43  Python                                 0x108fe142c _PyEval_EvalCode + 398
44  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
45  Python                                 0x108ef984c _PyObject_FastCallDictTstate + 215
46  Python                                 0x108f67213 slot_tp_call + 187
47  Python                                 0x108ef95ce _PyObject_MakeTpCall + 127
48  Python                                 0x108fe90a7 _PyEval_EvalFrameDefault + 27386
49  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
50  Python                                 0x108fe7ca4 _PyEval_EvalFrameDefault + 22263
51  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
52  Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
53  Python                                 0x108fe142c _PyEval_EvalCode + 398
54  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
55  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
56  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
57  Python                                 0x108efcb53 method_vectorcall + 160
58  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
59  Python                                 0x108fe142c _PyEval_EvalCode + 398
60  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
61  Python                                 0x108ef984c _PyObject_FastCallDictTstate + 215
62  Python                                 0x108f67213 slot_tp_call + 187
63  Python                                 0x108ef9e1a _PyObject_Call + 121
64  Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
65  Python                                 0x108fe142c _PyEval_EvalCode + 398
66  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
67  Python                                 0x108fe7957 _PyEval_EvalFrameDefault + 21418
68  Python                                 0x108fe142c _PyEval_EvalCode + 398
69  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
70  Python                                 0x108efcb53 method_vectorcall + 160
71  Python                                 0x108fe79bc _PyEval_EvalFrameDefault + 21519
72  Python                                 0x108fe142c _PyEval_EvalCode + 398
73  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
74  Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
75  Python                                 0x108fe142c _PyEval_EvalCode + 398
76  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
77  Python                                 0x108fe7957 _PyEval_EvalFrameDefault + 21418
78  Python                                 0x108fe142c _PyEval_EvalCode + 398
79  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
80  Python                                 0x108fe79bc _PyEval_EvalFrameDefault + 21519
81  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
82  Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
83  Python                                 0x108fe142c _PyEval_EvalCode + 398
84  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
85  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
86  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
87  Python                                 0x108efcb53 method_vectorcall + 160
88  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
89  Python                                 0x108fe142c _PyEval_EvalCode + 398
90  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
91  Python                                 0x108ef984c _PyObject_FastCallDictTstate + 215
92  Python                                 0x108f67213 slot_tp_call + 187
93  Python                                 0x108ef95ce _PyObject_MakeTpCall + 127
94  Python                                 0x108fe90a7 _PyEval_EvalFrameDefault + 27386
95  Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
96  Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
97  Python                                 0x108fe142c _PyEval_EvalCode + 398
98  Python                                 0x108efa078 _PyFunction_Vectorcall + 376
99  Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
100 Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
101 Python                                 0x108efcb53 method_vectorcall + 160
102 Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
103 Python                                 0x108fe142c _PyEval_EvalCode + 398
104 Python                                 0x108efa078 _PyFunction_Vectorcall + 376
105 Python                                 0x108ef984c _PyObject_FastCallDictTstate + 215
106 Python                                 0x108f67213 slot_tp_call + 187
107 Python                                 0x108ef95ce _PyObject_MakeTpCall + 127
108 Python                                 0x108fe90a7 _PyEval_EvalFrameDefault + 27386
109 Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
110 Python                                 0x108fe7957 _PyEval_EvalFrameDefault + 21418
111 Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
112 Python                                 0x108fe7957 _PyEval_EvalFrameDefault + 21418
113 Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
114 Python                                 0x108fe3295 _PyEval_EvalFrameDefault + 3304
115 Python                                 0x108fe142c _PyEval_EvalCode + 398
116 Python                                 0x108efa078 _PyFunction_Vectorcall + 376
117 Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
118 Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
119 Python                                 0x108efcb53 method_vectorcall + 160
120 Python                                 0x108fe8eff _PyEval_EvalFrameDefault + 26962
121 Python                                 0x108fe142c _PyEval_EvalCode + 398
122 Python                                 0x108efa078 _PyFunction_Vectorcall + 376
123 Python                                 0x108ef984c _PyObject_FastCallDictTstate + 215
124 Python                                 0x108f67213 slot_tp_call + 187
125 Python                                 0x108ef95ce _PyObject_MakeTpCall + 127
126 Python                                 0x108fe90a7 _PyEval_EvalFrameDefault + 27386
127 Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
128 Python                                 0x108fe7957 _PyEval_EvalFrameDefault + 21418
129 Python                                 0x108ef9fc0 _PyFunction_Vectorcall + 192
130 Python                                 0x108fe7957 _PyEval_EvalFrameDefault + 21418
131 Python                                 0x108fe142c _PyEval_EvalCode + 398
132 Python                                 0x10903c1de run_eval_code_obj + 128
133 Python                                 0x10903c136 run_mod + 96
134 Python                                 0x1090398f4 pyrun_file + 167
135 Python                                 0x1090392e4 pyrun_simple_file + 271
136 Python                                 0x1090391af PyRun_SimpleFileExFlags + 67
137 Python                                 0x109058bfb pymain_run_file + 326
138 Python                                 0x109058469 Py_RunMain + 993
139 Python                                 0x1090597c4 Py_BytesMain + 42
140 dyld                                   0x2006a551e start + 462

 

 

Sending the header lower-case, {}authorization{}, works. There is no indication that headers are case-sensitive and the error is most unhelpful at pointing to the underlying issue. If headers are not case-insensitive, I would at least hope for a better error diagnostic that would tell me so.

Environment: Python 3.9.12 on macOS 12.3.1
Reporter: Paul Horn
Assignee: David Li / @lidavidm

PRs and other links:

Note: This issue was originally created as ARROW-16606. Please see the migration documentation for further details.
@asfimport
Copy link
Collaborator Author

David Li / @lidavidm:
Thanks for the report. I'll try to reproduce it soon but yes, headers should absolutely be case insensitive and we shouldn't be crashing in either case.

@asfimport
Copy link
Collaborator Author

David Li / @lidavidm:
I guess your copy of gRPC doesn't report this, but the gRPC implementation used here requires lower case headers, and logs this:


E0519 09:08:26.158858765   23918 call.cc:779]                validate_metadata: {"created":"@1652965706.158849355","description":"Illegal header key","file":"/home/conda/feedstock_root/build_artifacts/grpc-cpp_1651922379710/work/src/core/lib/surface/validate_metadata.cc","file_line":55,"offset":0,"raw_bytes":"41 75 74 68 6f 72 69 7a 61 74 69 6f 6e 'Authorization'\u0000"}

Do you see this?

@asfimport
Copy link
Collaborator Author

Paul Horn:
I did not see that log message. For me the client just aborts with the first error block (Abort trap: 6). I don't see anything else on either the client or the server (which is a Java flight server).

I was thinking it might have been related to me running this on an M1 Mac under Rosetta, so I installed an ARM-native python and pyarrow, but I get the same error.

@asfimport
Copy link
Collaborator Author

David Li / @lidavidm:
Thanks for checking.

I'm thinking in Python we can lowercase the headers for you to avoid this (I don't want to add redundant checks on the C++ side), and document this more clearly in C++/Python. Is that reasonable?

@asfimport
Copy link
Collaborator Author

Paul Horn:
Yes, that sounds good, thank you!

@asfimport
Copy link
Collaborator Author

Antoine Pitrou / @pitrou:
Issue resolved by pull request 13274
#13274

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lidavidm lidavidm

Labels
Component: FlightRPC Component: Python Type: bug
Projects
None yet
Milestone
9.0.0
Development

No branches or pull requests
2 participants
@lidavidm @asfimport