-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intercept tls traffic from a python or node application #346
Comments
Python uses For more details, please refer to #336. |
Regarding the issue with Node, I will investigate it soon. Thank you for your feedback. |
@cfc4n thanks for sharing the python thread. My understanding is that currently we are probing the read/write calls made to the openssl lib and that makes it hard to map it to actual tcp request/response. You also mentioned it would become harder if the payload is too big, because multiple read/write calls would need to be made, making it harder to track the entire message. I’m curious whether concurrent requests may also become harder to map to the original request/response. I’ve read about a method in which we instead extract the keys and then perform decryption on the request/response payload. Any idea of advantages of such methods? |
Yes, concurrent requests can cause captured packets to be out of order. However, if it is the same HTTPS request and response that only triggers the pcapng mode can effectively avoid this kind of problem, but the disadvantage is also obvious. If eCapture is started after HTTPS key exchange is completed, it will not be able to capture the key and decryption cannot be completed. |
hello @slayerjain , any other quesions else? |
Hi @cfc4n this is more of a discussion question: can you please help me understand if this approach works on mapping which read/write call to openssl corresponds to which http req/resp transaction if there are concurrent requests? Also for node should we create a new issue? |
concurrent requestsI think eCapture's text mode cannot achieve true perfection, but if the network traffic in your environment is not particularly high, then the text mode can meet your needs. If the traffic is heavy, it is recommended to use pcapng mode. However, the disadvantage is that it cannot output in real-time and you need to save the network packets as a pcapng file before reading it. Noun explanation:
For nodea new issue needs to be created and I will follow up on it soon. |
I tested HTTPS requests with Python 3 and Node v10.19.0, on Ubuntu 20.04 (kernel 5.4), and eCapture was able to capture plaintext text without any issues. What is your testing environment? root@vm-server-2004:/home/cfc4n/project/ecapture# bin/ecapture tls
tls_2023/04/21 14:46:52 ECAPTURE :: ecapture Version : linux_x86_64:0.4.11-20230205-09197fd:5.4.0-131-generic
tls_2023/04/21 14:46:52 ECAPTURE :: Pid Info : 123102
tls_2023/04/21 14:46:52 ECAPTURE :: Kernel Info : 5.4.210
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL module initialization
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL master key keylogger: ecapture_masterkey.log
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL Module.Run()
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL UPROBE MODEL
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL origin version:OpenSSL 1.1.1f, as key:openssl 1.1.1f
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libssl.so.1.1
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL Hook masterKey function:SSL_write
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL target all process.
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL target all users.
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL BPF bytecode filename:user/bytecode/openssl_1_1_1d_kern.o
tls_2023/04/21 14:46:52 EBPFProbeOPENSSL module started successfully.
tls_2023/04/21 14:46:52 EBPFProbeGNUTLS module initialization
tls_2023/04/21 14:46:52 EBPFProbeGNUTLS Module.Run()
tls_2023/04/21 14:46:52 EBPFProbeGNUTLS BPF bytecode filename:user/bytecode/gnutls_kern.o
tls_2023/04/21 14:46:52 EBPFProbeGNUTLS HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libgnutls.so.30
tls_2023/04/21 14:46:52 EBPFProbeGNUTLS target all process.
tls_2023/04/21 14:46:52 EBPFProbeGNUTLS module started successfully.
tls_2023/04/21 14:46:52 EBPFProbeNSPR module initialization
tls_2023/04/21 14:46:52 EBPFProbeNSPR Module.Run()
tls_2023/04/21 14:46:52 EBPFProbeNSPR BPF bytecode filename:user/bytecode/nspr_kern.o
tls_2023/04/21 14:46:52 EBPFProbeNSPR HOOK type:2, binrayPath:/lib/x86_64-linux-gnu/libnspr4.so
tls_2023/04/21 14:46:52 EBPFProbeNSPR target all process.
tls_2023/04/21 14:46:52 EBPFProbeNSPR module started successfully.
tls_2023/04/21 14:46:52 ECAPTURE :: start 3 modules
tls_2023/04/21 14:46:59 TLS1_2_VERSION: save CLIENT_RANDOM ec9a538b6f128ce4aa2aa86993d6b846d0eb4640ec6c73e83f7348611fb9eae5 to file success, 176 bytes
tls_2023/04/21 14:47:00 eventWorker TickerCount > 10, event closed.
tls_2023/04/21 14:47:00 UUID:123135_123135_python3_3_1, Name:HTTPRequest, Type:1, Length:135
tls_2023/04/21 14:47:00
GET / HTTP/1.1
Host: www.baidu.com
Connection: close
Accept-Encoding: identity
Connection: close
User-Agent: Python-urllib/3.8
tls_2023/04/21 14:47:00 eventWorker TickerCount > 10, event closed.
2023/04/21 14:47:00 HTTPS Headers
X-Ua-Compatible => [IE=Edge,chrome=1]
P3p => [CP=" OTI DSP COR IVA OUR IND COM " CP=" OTI DSP COR IVA OUR IND COM "]
Pragma => [no-cache]
Strict-Transport-Security => [max-age=0]
Traceid => [1682088419064668877810287991152751756197]
Accept-Ranges => [bytes]
Content-Length => [227]
Content-Security-Policy => [frame-ancestors 'self' https://chat.baidu.com https://fj-chat.baidu.com https://hba-chat.baidu.com https://hbe-chat.baidu.com https://njjs-chat.baidu.com https://nj-chat.baidu.com https://hna-chat.baidu.com https://hnb-chat.baidu.com;]
Content-Type => [text/html]
Server => [BWS/1.1]
Set-Cookie => [BD_NOT_HTTPS=1; path=/; Max-Age=300 BIDUPSID=B4512D374E85E5238A289AA66A7EF9DF; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com PSTM=1682088419; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com BAIDUID=B4512D374E85E523C59FDB99AFB7DF02:FG=1; max-age=31536000; expires=Sat, 20-Apr-24 14:46:59 GMT; domain=.baidu.com; path=/; version=1; comment=bd]
Cache-Control => [no-cache]
Date => [Fri, 21 Apr 2023 14:46:59 GMT]
tls_2023/04/21 14:47:00 UUID:123135_123135_python3_3_0, Name:HTTPResponse, Type:3, Length:1362
tls_2023/04/21 14:47:00
HTTP/1.1 200 OK
Connection: close
Content-Length: 227
Accept-Ranges: bytes
Cache-Control: no-cache
Content-Security-Policy: frame-ancestors 'self' https://chat.baidu.com https://fj-chat.baidu.com https://hba-chat.baidu.com https://hbe-chat.baidu.com https://njjs-chat.baidu.com https://nj-chat.baidu.com https://hna-chat.baidu.com https://hnb-chat.baidu.com;
Content-Type: text/html
Date: Fri, 21 Apr 2023 14:46:59 GMT
P3p: CP=" OTI DSP COR IVA OUR IND COM "
P3p: CP=" OTI DSP COR IVA OUR IND COM "
Pragma: no-cache
Server: BWS/1.1
Set-Cookie: BD_NOT_HTTPS=1; path=/; Max-Age=300
Set-Cookie: BIDUPSID=B4512D374E85E5238A289AA66A7EF9DF; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: PSTM=1682088419; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BAIDUID=B4512D374E85E523C59FDB99AFB7DF02:FG=1; max-age=31536000; expires=Sat, 20-Apr-24 14:46:59 GMT; domain=.baidu.com; path=/; version=1; comment=bd
Strict-Transport-Security: max-age=0
Traceid: 1682088419064668877810287991152751756197
X-Ua-Compatible: IE=Edge,chrome=1
<html>
<head>
<script>
location.replace(location.href.replace("https://","http://"));
</script>
</head>
<body>
<noscript><meta http-equiv="refresh" content="0;url=http://www.baidu.com/"></noscript>
</body>
</html>
tls_2023/04/21 14:47:06 TLS1_2_VERSION: save CLIENT_RANDOM 00d9f710436ab893ea878d7b36d4e5d2ea5ce9156242e93199cd8de861a5e054 to file success, 176 bytes
tls_2023/04/21 14:47:06 UUID:123204_123204_node_0_0, Name:DefaultParser, Type:0, Length:1179
tls_2023/04/21 14:47:06
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: no-cache
Content-Length: 9508
Content-Security-Policy: frame-ancestors 'self' https://chat.baidu.com https://fj-chat.baidu.com https://hba-chat.baidu.com https://hbe-chat.baidu.com https://njjs-chat.baidu.com https://nj-chat.baidu.com https://hna-chat.baidu.com https://hnb-chat.baidu.com;
Content-Type: text/html
Date: Fri, 21 Apr 2023 14:47:06 GMT
P3p: CP=" OTI DSP COR IVA OUR IND COM "
P3p: CP=" OTI DSP COR IVA OUR IND COM "
Pragma: no-cache
Server: BWS/1.1
Set-Cookie: BAIDUID=CEE530010618FC7920473FE3F1DB55CA:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BIDUPSID=CEE530010618FC7920473FE3F1DB55CA; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: PSTM=1682088426; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BAIDUID=CEE530010618FC79A4397C1C3B45BCC7:FG=1; max-age=31536000; expires=Sat, 20-Apr-24 14:47:06 GMT; domain=.baidu.com; path=/; version=1; comment=bd
Traceid: 168208842606074982508145908752426775467
Vary: Accept-Encoding
X-Ua-Compatible: IE=E
tls_2023/04/21 14:47:06 UUID:123204_123204_node_0_0, Name:DefaultParser, Type:0, Length:547
tls_2023/04/21 14:47:06
dge,chrome=1
Connection: close
<!DOCTYPE html><html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="description" content="全球领先的中文搜索引擎、致力于让网民更便捷地获取信息,找到所求。百度超过千亿的中文网页数据库,可以瞬间找到相关的搜索结果。"><link rel="shortcut icon" href="//www.baidu.com/favicon.ico" type="image/x-icon"><link rel="search
tls_2023/04/21 14:47:06 UUID:123204_123204_node_0_0, Name:DefaultParser, Type:0, Length:3537
tls_2023/04/21 14:47:06
" type="application/opensearchdescription+xml" href="//www.baidu.com/content-search.xml" title="百度搜索"><title>百度一下,你就知道</title><style type="text/css">body{margin:0;padding:0;text-align:center;background:#fff;height:100%}html{overflow-y:auto;color:#000;overflow:-moz-scrollbars;height:100%}body,input{font-size:12px;font-family:"PingFang SC",Arial,"Microsoft YaHei",sans-serif}a{text-decoration:none}a:hover{text-decoration:underline}img{border:0;-ms-interpolation-mode:bicubic}input{font-size:100%;border:0}body,form{position:relative;z-index:0}#wrapper{height:100%}#head_wrapper.s-ps-islite{padding-bottom:370px}#head_wrapper.s-ps-islite .s_form{position:relative;z-index:1}#head_wrapper.s-ps-islite .fm{position:absolute;bottom:0}#head_wrapper.s-ps-islite .s-p-top{position:absolute;bottom:40px;width:100%;height:181px}#head_wrapper.s-ps-islite #s_lg_img{position:static;margin:33px auto 0 auto;left:50%}#form{z-index:1}.s_form_wrapper{height:100%}#lh{margin:16px 0 5px;word-spacing:3px}.c-font-normal{font:13px/23px Arial,sans-serif}.c-color-t{color:#222}.c-btn,.c-btn:visited{color:#333!important}.c-btn{display:inline-block;overflow:hidden;font-family:inherit;font-weight:400;text-align:center;vertical-align:middle;outline:0;border:0;height:30px;width:80px;line-height:30px;font-size:13px;border-radius:6px;padding:0;background-color:#f5f5f6;cursor:pointer}.c-btn:hover{background-color:#315efb;color:#fff!important}a.c-btn{text-decoration:none}.c-btn-mini{height:24px;width:48px;line-height:24px}.c-btn-primary,.c-btn-primary:visited{color:#fff!important}.c-btn-primary{background-color:#4e6ef2}.c-btn-primary:hover{background-color:#315efb}a:active{color:#f60}#wrapper{position:relative;min-height:100%}#head{padding-bottom:100px;text-align:center}#wrapper{min-width:1250px;height:100%;min-height:600px}#head{position:relative;padding-bottom:0;height:100%;min-height:600px}.s_form_wrapper{height:100%}.quickdelete-wrap{position:relative}.tools{position:absolute;right:-75px}.s-isindex-wrap{position:relative}#head_wrapper.head_wrapper{width:auto}#head_wrapper{position:relative;height:40%;min-height:314px;max-height:510px;width:1000px;margin:0 auto}#head_wrapper .s-p-top{height:60%;min-height:185px;max-height:310px;position:relative;z-index:0;text-align:center}#head_wrapper input{outline:0;-webkit-appearance:none}#head_wrapper .s_btn_wr,#head_wrapper .s_ipt_wr{display:inline-block;zoom:1;background:0 0;vertical-align:top}#head_wrapper .s_ipt_wr{position:relative;width:546px}#head_wrapper .s_btn_wr{width:108px;height:44px;position:relative;z-index:2}#head_wrapper .s_ipt_wr:hover #kw{border-color:#a7aab5}#head_wrapper #kw{width:512px;height:16px;padding:12px 16px;font-size:16px;margin:0;vertical-align:top;outline:0;box-shadow:none;border-radius:10px 0 0 10px;border:2px solid #c4c7ce;background:#fff;color:#222;overflow:hidden;box-sizing:content-box}#head_wrapper #kw:focus{border-color:#4e6ef2!important;opacity:1}#head_wrapper .s_form{width:654px;height:100%;margin:0 auto;text-align:left;z-index:100}#head_wrapper .s_btn{cursor:pointer;width:108px;height:44px;line-height:45px;padding:0;background:0 0;background-color:#4e6ef2;border-radius:0 10px 10px 0;font-size:17px;color:#fff;box-shadow:none;font-weight:400;border:none;outline:0}#head_wrapper .s_btn:hover{background-color:#4662d9}#head_wrapper .s_btn:active{background-color:#4662d9}#head_wrapper .quickdelete-wrap{position:relative}#s_top_wrap{position:absolute;z-index:99;min-width:1000px;width:100%}.s-top-left{position:absolute;l
tls_2023/04/21 14:47:06 UUID:123204_123204_node_0_0, Name:DefaultParser, Type:0, Length:4096
tls_2023/04/21 14:47:06
eft:0;top:0;z-index:100;height:60px;padding-left:24px}.s-top-left .mnav{margin-right:31px;margin-top:19px;display:inline-block;position:relative}.s-top-left .mnav:hover .s-bri,.s-top-left a:hover{color:#315efb;text-decoration:none}.s-top-left .s-top-more-btn{padding-bottom:19px}.s-top-left .s-top-more-btn:hover .s-top-more{display:block}.s-top-right{position:absolute;right:0;top:0;z-index:100;height:60px;padding-right:24px}.s-top-right .s-top-right-text{margin-left:32px;margin-top:19px;display:inline-block;position:relative;vertical-align:top;cursor:pointer}.s-top-right .s-top-right-text:hover{color:#315efb}.s-top-right .s-top-login-btn{display:inline-block;margin-top:18px;margin-left:32px;font-size:13px}.s-top-right a:hover{text-decoration:none}#bottom_layer{width:100%;position:fixed;z-index:302;bottom:0;left:0;height:39px;padding-top:1px;overflow:hidden;zoom:1;margin:0;line-height:39px;background:#fff}#bottom_layer .lh{display:inline;margin-right:20px}#bottom_layer .lh:last-child{margin-left:-2px;margin-right:0}#bottom_layer .lh.activity{font-weight:700;text-decoration:underline}#bottom_layer a{font-size:12px;text-decoration:none}#bottom_layer .text-color{color:#bbb}#bottom_layer a:hover{color:#222}#bottom_layer .s-bottom-layer-content{text-align:center}</style></head><body><div id="wrapper" class="wrapper_new"><div id="head"><div id="s-top-left" class="s-top-left s-isindex-wrap"><a href="//news.baidu.com/" target="_blank" class="mnav c-font-normal c-color-t">新闻</a><a href="//www.hao123.com/" target="_blank" class="mnav c-font-normal c-color-t">hao123</a><a href="//map.baidu.com/" target="_blank" class="mnav c-font-normal c-color-t">地图</a><a href="//live.baidu.com/" target="_blank" class="mnav c-font-normal c-color-t">直播</a><a href="//haokan.baidu.com/?sfrom=baidu-top" target="_blank" class="mnav c-font-normal c-color-t">视频</a><a href="//tieba.baidu.com/" target="_blank" class="mnav c-font-normal c-color-t">贴吧</a><a href="//xueshu.baidu.com/" target="_blank" class="mnav c-font-normal c-color-t">学术</a><div class="mnav s-top-more-btn"><a href="//www.baidu.com/more/" name="tj_briicon" class="s-bri c-font-normal c-color-t" target="_blank">更多</a></div></div><div id="u1" class="s-top-right s-isindex-wrap"><a class="s-top-login-btn c-btn c-btn-primary c-btn-mini lb" style="position:relative;overflow:visible" name="tj_login" href="//www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1">登录</a></div><div id="head_wrapper" class="head_wrapper s-isindex-wrap s-ps-islite"><div class="s_form"><div class="s_form_wrapper"><div id="lg" class="s-p-top"><img hidefocus="true" id="s_lg_img" class="index-logo-src" src="//www.baidu.com/img/flexible/logo/pc/index.png" width="270" height="129" usemap="#mp"><map name="mp"><area style="outline:0" hidefocus="true" shape="rect" coords="0,0,270,129" href="//www.baidu.com/s?wd=%E7%99%BE%E5%BA%A6%E7%83%AD%E6%90%9C&sa=ire_dl_gh_logo_texing&rsv_dl=igh_logo_pcs" target="_blank" title="点击一下,了解更多"></map></div><a href="//www.baidu.com/" id="result_logo"></a><form id="form" name="f" action="//www.baidu.com/s" class="fm"><input type="hidden" name="ie" value="utf-8"> <input type="hidden" name="f" value="8"> <input type="hidden" name="rsv_bp" value="1"> <input type="hidden" name="rsv_idx" value="1"> <input type="hidden" name="ch" value=""> <input type="hidden" name="tn" value="baidu"> <input type="hidden" name="bar" value=""> <span class="s_ipt_wr quickdelete-wrap"><input id="kw" name="wd" class="s_ipt" value="" maxlength="255" autocomplete="off"> </span><span class="s_btn_wr"><input type="submit" id="su" value="百度一下" class="bg s_btn"> </span><input type="hidden" name="rn" value=""> <input type="hidden" name="fenlei" value="256"> <input type="hidden" name="oq" value=""> <input type="hidden" name="rsv_pq" value="b9ff093e0000e419"> <input type="hidden" name="rsv_t" value="3635FYbdbC8tlWmudZmYaUnaucNe+RzTzNEGqg/JuniQU10WL5mtMQehIrU"> <input type="hidden" name="rqlang" value="cn"> <input type="hidden" n
tls_2023/04/21 14:47:06 UUID:123204_123204_node_0_0, Name:DefaultParser, Type:0, Length:743
tls_2023/04/21 14:47:06
-color" href="//www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000001" target="_blank">京公网安备11000002000001号</a></p><p class="lh"><a class="text-color" href="//beian.miit.gov.cn/" target="_blank">京ICP证030173号</a></p><p class="lh"><span id="year" class="text-color"></span></p><p class="lh"><span class="text-color">互联网药品信息服务资格证书 (京)-经营性-2017-0020</span></p><p class="lh"><a class="text-color" href="//www.baidu.com/licence/" target="_blank">信息网络传播视听节目许可证 0110516</a></p></div></div></div></div><script type="text/javascript">var date=new Date,year=date.getFullYear();document.getElementById("year").innerText="©"+year+" Baidu "</script></body></html>
tls_2023/04/21 14:47:07 eventWorker TickerCount > 10, event closed.
tls_2023/04/21 14:47:07 UUID:123204_123204_node_0_1, Name:HTTPRequest, Type:1, Length:77
tls_2023/04/21 14:47:07
GET / HTTP/1.1
Host: www.baidu.com
Connection: close
Connection: close
tls_2023/04/21 14:47:07 eventWorker TickerCount > 10, event closed. open another shell ,and run command cfc4n@vm-server-2004:~/project/ecapture/tests
node node_https.js
python3 python3_https.py refer #349 |
ping @slayerjain |
Hi @cfc4n , my environment was a ubuntu 22.04 VM (with kernel 5.15) running in M1 (arm64) macbook via patallels. There could be a mistake on my end. I’ll check and report back today. I’m sorry for delayed response. |
@cfc4n so I tried this but for node it only worked the first time and didn't work after that. I tried switching the url to google.com or bing.com, but no luck. I used the python and node programs from your test files.
Python version: 3.10.6 am I supposed to use a different binary? |
Can you give me your SSH account? I want to debug on your host. my email : Y2ZjNG54QGdtYWlsLmNvbQ== |
No response, closed. |
Describe the bug
I tried running
sudo ecapture -tls
and it was able to capture tls traffic from curl but it didn't work for traffic from node or python application.To Reproduce
Steps to reproduce the behavior:
sudo ./ecapture tls
Expected behavior
ecapture
should be able to capture the traffic because I think node and python also use openssl.Additional context
N/A
The text was updated successfully, but these errors were encountered: