Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature request] Data streaming #32

Closed
acheong08 opened this issue Feb 9, 2023 · 3 comments
Closed

[Feature request] Data streaming #32

acheong08 opened this issue Feb 9, 2023 · 3 comments

Comments

@acheong08
Copy link

Similar to Python requests: https://stackoverflow.com/questions/60343944/how-does-requests-stream-true-option-streams-data-one-block-at-a-time

@GunGunGun
Copy link

GunGunGun commented Feb 25, 2023

Streaming would be great to have because it allows user to make decision about skipping body data or buffer fetching to print result to client, which is great in many situations:

  • Content is too big, like 2GB and memory limitation like 2GB RAM, then if you fetch everything into memory you'll get OOM error
  • In proxy environment, you may want to abort the body data if the response header doesn't match some of your criterias

I'll try to look into the code and implement this feature, shouldn't be that hard considering we're fetching data from Go TLS Client which probably allow data streaming.

This line going forward should be what we need to modify: https://github.com/FlorianREGAZ/Python-Tls-Client/blob/master/tls_client/response.py#L40

@GunGunGun
Copy link

GunGunGun commented Feb 25, 2023

Actually after researching the API of tls-client by bogdanfinn, I can't find any information about stream being made available, so it looks like to have this stream feature we first need to do something with the core first, which is the Go Client https://github.com/bogdanfinn/tls-client-api

And... tls-client actually sends back JSON data instead of standard HTTP response with headers \r\n\r\n then body, which is ways easier to handle streaming (the same way requests and urllib3 are doing). Something like this, which is pretty pointless to stream data if you want to send buffer to something like web browsers, in this case you need to parse JSON data anyways, then send to web browsers:

{....,"headers":{"Access-Control-Allow-Credentials":["true"],"Access-Control-Allow-Origin":["https://rr6-sn-8qj-i5o6k.gooqlevideo.com"],"Age":["72715"],"Alt-Svc":["h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"],"Cache-Control":["max-age=31536000"],"Cf-Cache-Status":["HIT"],"Cf-Ray":["79f028e9cf569e28-SIN"],"Content-Encoding":["br"],"Content-Security-Policy":["default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;form-action 'self';frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests"],"Content-Type":["text/html; charset=utf-8"],"Cross-Origin-Embedder-Policy":["require-corp"],"Cross-Origin-Opener-Policy":["same-origin"],"Date":["Sat, 25 Feb 2023 11:39:19 GMT"],"Expect-Ct":["max-age=0"],"Origin-Agent-Cluster":["?1"],"Referrer-Policy":["no-referrer"],"Server":["cloudflare"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"],"Vary":["Accept-Encoding","Origin"],"X-Content-Type-Options":["nosniff"],"X-Dns-Prefetch-Control":["off"],"X-Download-Options":["noopen"],"X-Frame-Options":["SAMEORIGIN"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Powered-By":["centminmod"],"X-Xss-Protection":["0","1; mode=block"]},"cookies":{"__cf_bm":"yxyOVhUi38n1UfQujprACZl5yxXyhYZMYl6boMzKKh0-1677324536-0-AdwLPV4avEeaFZNNLGA7VKzwHM3C2rbm2jL/SPwv1WNoRDaahAKij0hWwlApbK52ErrqT3pGG+pCYWIZlOzkM9o="}}'

@FlorianREGAZ
Copy link
Owner

Hey, I don't think it'll be possible to add this soon sadly. :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants