Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new reddit api changes #12

Closed
asad-awadia opened this issue Jul 28, 2023 · 8 comments
Closed

new reddit api changes #12

asad-awadia opened this issue Jul 28, 2023 · 8 comments

Comments

@asad-awadia
Copy link

with the new reddit api changes is this still working as expected?

@asad-awadia
Copy link
Author

@torfsen

@torfsen
Copy link
Owner

torfsen commented Jul 29, 2023

Well, are you running into any particular issues?

By the way, there is no need to explicitly mention the owner of a repo using @ when you create an issue -- they get notified anyways, and also pinging them on top of that just creates noise.

@asad-awadia
Copy link
Author

yeah - the stream returns only errors

error decoding response body: missing field data at line 1 column 46 is the error but not sure why

@torfsen
Copy link
Owner

torfsen commented Aug 1, 2023

Which version of the library are you using?

The provided example works for me as expected. Does it work for you?

Please post a minimal, reproducible example of the code that is not working for you.

@asad-awadia
Copy link
Author

0.2.0 of roux-stream and 2.2.7 of roux

No it does not work for me - the provided example also has the same issue. It doesn't have an if comment.is_err() check - but if I add that then the same error can be seen which is missing field data", line: 1, column: 46

right now this is the rust backtrace

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: SourceError(Network(reqwest::Error { kind: Decode, source: Error("missing field `data`", line: 1, column: 46) }))', src/main.rs:232:24
stack backtrace:
   0:        0x1063d3fd6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h6ae6bff14318b704
   1:        0x1063e94da - core::fmt::write::h072391c2938df059
   2:        0x1063b838c - std::io::Write::write_fmt::h88fcf164d5e43745
   3:        0x1063d3db0 - std::sys_common::backtrace::print::h335d3fd0c103876b
   4:        0x1063be413 - std::panicking::default_hook::{{closure}}::hd84d4fd3324d0128
   5:        0x1063be102 - std::panicking::default_hook::h141bffb40908b0d1
   6:        0x1063bec81 - std::panicking::rust_panic_with_hook::h770b31c549b79e3e
   7:        0x1063d46e4 - std::panicking::begin_panic_handler::{{closure}}::hddf7da8913935775
   8:        0x1063d4138 - std::sys_common::backtrace::__rust_end_short_backtrace::haa61a482c4c687d9
   9:        0x1063be7dd - _rust_begin_unwind
  10:        0x1063fbb43 - core::panicking::panic_fmt::h208cfee7b750b4aa
  11:        0x1063fbfb5 - core::result::unwrap_failed::h65874989211a1511
  12:        0x105e613dd - core::result::Result<T,E>::unwrap::h564a3e6f54094ab0
                               at /private/tmp/rust-20230331-7129-10kjp3b/rustc-1.68.2-src/library/core/src/result.rs:1112:23
  13:        0x105e4626c - foo::main::{{closure}}::h444735cb282cd929
                               at /Users/asadawadia/Documents/dev/rust/src/main.rs:232:6
  14:        0x105e3c7f2 - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::h9f69fc39eb2a1840
                               at /Users/asadawadia/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:63
  15:        0x105e3c696 - tokio::runtime::coop::with_budget::h6d6bd76ea24b0388
                               at /Users/asadawadia/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:102:5
  16:        0x105e3c696 - tokio::runtime::coop::budget::hb009c17bdf9ebbbd
                               at /Users/asadawadia/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/coop.rs:68:5
  17:        0x105e3c696 - tokio::runtime::park::CachedParkThread::block_on::h6ddf560c642b94c3
                               at /Users/asadawadia/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/park.rs:283:31
  18:        0x105e58aa1 - tokio::runtime::context::BlockingRegionGuard::block_on::hb530a3622a4e0861
                               at /Users/asadawadia/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/context.rs:295:13
  19:        0x105e4df9f - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h0b4e7467f3381bc3
                               at /Users/asadawadia/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/scheduler/multi_thread/mod.rs:66:9
  20:        0x105e4132f - tokio::runtime::runtime::Runtime::block_on::h01d92aa76cbf45ef
                               at /Users/asadawadia/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.23.0/src/runtime/runtime.rs:284:45
  21:        0x105e3fed3 -foo::main::h5394f5c2156a607f
                               at /Users/asadawadia/Documents/dev/rust/src/main.rs:235:6
  22:        0x105e541de - core::ops::function::FnOnce::call_once::h0db3ea3feca9aaca
                               at /private/tmp/rust-20230331-7129-10kjp3b/rustc-1.68.2-src/library/core/src/ops/function.rs:250:5
  23:        0x105e48de1 - std::sys_common::backtrace::__rust_begin_short_backtrace::h8229d0b1c80e2bcc
                               at /private/tmp/rust-20230331-7129-10kjp3b/rustc-1.68.2-src/library/std/src/sys_common/backtrace.rs:121:18
  24:        0x105e445e4 - std::rt::lang_start::{{closure}}::h992552f10b5eb053
                               at /private/tmp/rust-20230331-7129-10kjp3b/rustc-1.68.2-src/library/std/src/rt.rs:166:18
  25:        0x1063bc964 - std::rt::lang_start_internal::h2789a7315774cae2
  26:        0x105e445b7 - std::rt::lang_start::ha140bf980a9a39e3
                               at /private/tmp/rust-20230331-7129-10kjp3b/rustc-1.68.2-src/library/std/src/rt.rs:165:17
  27:        0x105e3ff58 - _main
  28:     0x7ff80937c310 - <unknown>

@torfsen
Copy link
Owner

torfsen commented Aug 14, 2023

The provided example works for me as expected. Does it work for you?

No it does not work for me - the provided example also has the same issue. It doesn't have an if comment.is_err() check - but if I add that then the same error can be seen [...]

The example does include a check for an error value in comment, so I'm not sure why adding an additional is_err check would change anything.

Again, please a minimal, reproducible example of the code that is not working for you. Otherwise I won't be able to look into this any further.

@asad-awadia
Copy link
Author

@torfsen

please a minimal, reproducible example of the code

here you go - https://github.com/asad-awadia/reproducer

SourceError(Network(reqwest::Error { kind: Decode, source: Error("missing field data", line: 1, column: 46) })) the errors

so I'm not sure why adding an additional is_err

The error check does not actually print the errors so they probably go unnoticed

@torfsen
Copy link
Owner

torfsen commented Aug 26, 2023

I've finally been able to reproduce the issue. The example you provided still was rather complex, in the future please try to provide a minimal example. Here's what I've condensed your code to:

use futures_util::*;
use roux_stream::stream_comments;
use std::io;
use std::io::Write;
use std::time::Duration;
use tokio_retry::strategy::ExponentialBackoff;

#[tokio::main]
async fn main() {
    let subreddit = roux::subreddit::Subreddit::new("Guitar");

    let retry_strategy = ExponentialBackoff::from_millis(5).factor(100).take(3);

    let (mut stream, join_handle) = stream_comments(
        &subreddit,
        Duration::from_secs(2),
        retry_strategy,
        Some(Duration::from_secs(30)),
    );

    while let Some(comment) = stream.next().await {
        if comment.is_err() {
            panic!("{}", comment.err().unwrap().to_string())
        }
        print!(".");
        io::stdout().flush().unwrap();
    }

    join_handle.await.unwrap().unwrap();
}

Running

$ RUST_BACKTRACE=full cargo run

results in

    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
warning: the following packages contain code that will be rejected by a future version of Rust: rustc-serialize v0.3.24
note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
     Running `target/debug/reproducer-reddit-comments`
.........................................................................................................thread 'main' panicked at 'error decoding response body: missing field `data` at line 1 column 46', src/main.rs:23:13
stack backtrace:
   0:     0x558e8d80ee5a - std::backtrace_rs::backtrace::libunwind::trace::ha9053a9a07ca49cb
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x558e8d80ee5a - std::backtrace_rs::backtrace::trace_unsynchronized::h9c2852a457ad564e
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x558e8d80ee5a - std::sys_common::backtrace::_print_fmt::h457936fbfaa0070f
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x558e8d80ee5a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h5779d7bf7f70cb0c
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x558e8d830b9e - core::fmt::write::h5a4baaff1bcd3eb5
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/fmt/mod.rs:1232:17
   5:     0x558e8d80c0a5 - std::io::Write::write_fmt::h4bc1f301cb9e9cce
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/io/mod.rs:1684:15
   6:     0x558e8d80ec25 - std::sys_common::backtrace::_print::h5fcdc36060f177e8
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x558e8d80ec25 - std::sys_common::backtrace::print::h54ca9458b876c8bf
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x558e8d8103df - std::panicking::default_hook::{{closure}}::hbe471161c7664ed6
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:271:22
   9:     0x558e8d81011b - std::panicking::default_hook::ha3500da57aa4ac4f
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:290:9
  10:     0x558e8d810988 - std::panicking::rust_panic_with_hook::h50c09d000dc561d2
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:692:13
  11:     0x558e8d810889 - std::panicking::begin_panic_handler::{{closure}}::h9e2b2176e00e0d9c
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:583:13
  12:     0x558e8d80f2c6 - std::sys_common::backtrace::__rust_end_short_backtrace::h5739b8e512c09d02
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:150:18
  13:     0x558e8d810592 - rust_begin_unwind
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
  14:     0x558e8d34dda3 - core::panicking::panic_fmt::hf33a1475b4dc5c3e
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
  15:     0x558e8d36941c - core::panicking::panic_display::h0463bb9c4536c56f
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:147:5
  16:     0x558e8d35199d - reproducer_reddit_comments::main::{{closure}}::he0a68f2582a6fe96
                               at /home/torf/tmp/reproducer/src/main.rs:23:13
  17:     0x558e8d366c94 - tokio::runtime::park::CachedParkThread::block_on::{{closure}}::hca5f58925c996597
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/park.rs:282:63
  18:     0x558e8d36661c - tokio::runtime::coop::with_budget::hf5735b487dcc7897
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/coop.rs:107:5
  19:     0x558e8d36661c - tokio::runtime::coop::budget::h81f641a91284d602
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/coop.rs:73:5
  20:     0x558e8d36661c - tokio::runtime::park::CachedParkThread::block_on::ha075c65cf049aa57
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/park.rs:282:31
  21:     0x558e8d35b5ca - tokio::runtime::context::blocking::BlockingRegionGuard::block_on::hd69ed5c2b86d95b0
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/context/blocking.rs:66:9
  22:     0x558e8d3625d0 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}::hcaccc9a7130c9a82
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  23:     0x558e8d36dc4e - tokio::runtime::context::runtime::enter_runtime::heeb02cb2b789d773
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/context/runtime.rs:65:16
  24:     0x558e8d362598 - tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h6c49a10216ed117f
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  25:     0x558e8d351cb6 - tokio::runtime::runtime::Runtime::block_on::hc1d6e938ec78964f
                               at /home/torf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.31.0/src/runtime/runtime.rs:349:45
  26:     0x558e8d35967f - reproducer_reddit_comments::main::h207782b11d7eb69d
                               at /home/torf/tmp/reproducer/src/main.rs:21:5
  27:     0x558e8d36405b - core::ops::function::FnOnce::call_once::h792340cda8636e66
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/ops/function.rs:250:5
  28:     0x558e8d3597be - std::sys_common::backtrace::__rust_begin_short_backtrace::h5c60ff6352005886
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/sys_common/backtrace.rs:134:18
  29:     0x558e8d366d01 - std::rt::lang_start::{{closure}}::h98739cd100da83c0
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:166:18
  30:     0x558e8d80826c - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hd6efcd3bec896f2c
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/ops/function.rs:287:13
  31:     0x558e8d80826c - std::panicking::try::do_call::hce04e543bb1f4cbb
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:487:40
  32:     0x558e8d80826c - std::panicking::try::h3342dd4e1f680968
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:451:19
  33:     0x558e8d80826c - std::panic::catch_unwind::h148ce1e59ac0cee7
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panic.rs:140:14
  34:     0x558e8d80826c - std::rt::lang_start_internal::{{closure}}::h25f9dda2057a67fe
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:148:48
  35:     0x558e8d80826c - std::panicking::try::do_call::h7caaaeaf9401650b
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:487:40
  36:     0x558e8d80826c - std::panicking::try::he7d15285746cbbc2
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:451:19
  37:     0x558e8d80826c - std::panic::catch_unwind::h89fb4f50c0301fe0
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panic.rs:140:14
  38:     0x558e8d80826c - std::rt::lang_start_internal::h078acd489417d3c1
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:148:20
  39:     0x558e8d366cda - std::rt::lang_start::h86470c21a10e18b6
                               at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/rt.rs:165:17
  40:     0x558e8d35972e - main
  41:     0x7fbf9a984083 - __libc_start_main
                               at /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16
  42:     0x558e8d34e43e - _start
  43:                0x0 - <unknown>

Unfortunately, that backtrace isn't too helpful -- it doesn't really point us to where things go wrong. Following the calls "manually", we see that the issue is the decoding of the API response requesting the latest comments. The API URL for that is https://www.reddit.com/r/Guitar/comments.json. Querying it manually after the panic shows the real problem:

$ http 'https://www.reddit.com/r/Guitar/comments.json'
HTTP/1.1 429 Too Many Requests
Accept-Ranges: bytes
Cache-Control: private, max-age=3600
Connection: keep-alive
Content-Length: 46
Date: Sat, 26 Aug 2023 13:57:02 GMT
NEL: {"report_to": "w3-reporting-nel", "max_age": 14400, "include_subdomains": false, "success_fraction": 1.0, "failure_fraction": 1.0}
Report-To: {"group": "w3-reporting-nel", "max_age": 14400, "include_subdomains": true,  "endpoints": [{ "url": "https://w3-reporting-nel.reddit.com/reports" }]}, {"group": "w3-reporting", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting.reddit.com/reports" }]}, {"group": "w3-reporting-csp", "max_age": 14400, "include_subdomains": true, "endpoints": [{ "url": "https://w3-reporting-csp.reddit.com/reports" }]}
Server: snooserv
Set-Cookie: loid=000000000ieeynq7nu.2.1693058222192.Z0FBQUFBQms2Z1N1anhnd1VQZFd0WnY2eUt4alhYSkdEWUp6WDBrbUlDRWY5SGcweHVCeHZxb3VqT2xqTjB5c0pjTmtfdFFEbHNrZzcza1lFX2tfTkFNNU5VNW1DY1NUbzZwb2liMndWckFmN2NNMGpmYXBoUGd3WC04S1pPRnd2cmd4UWtiZzNFZW8; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Mon, 25-Aug-2025 13:57:02 GMT; secure; SameSite=None; Secure
Set-Cookie: session_tracker=hlnidpdbqfamhcmmqc.0.1693058222195.Z0FBQUFBQms2Z1N1d00wU1J5em55NWNsYUlFUW05OW00d2ZOUlRVN0ltdDF2b2N0S0R3ai0tbWdiSHgxSDBhZjNXdDJYN1hIR3E0YTh6dm5ub0hnakVxNUs5V1NUd0tyX0xCT3J5ZF8wR0ppM1k2WjlETG5TcGs0ZElKTnY4Z2szQUZyNFc1SHk3d2k; Domain=reddit.com; Max-Age=7199; Path=/; expires=Sat, 26-Aug-2023 15:57:02 GMT; secure; SameSite=None; Secure
Set-Cookie: csv=2; Max-Age=63072000; Domain=.reddit.com; Path=/; Secure; SameSite=None
Set-Cookie: edgebucket=0o7mLHqv0PFDmUoBkI; Domain=reddit.com; Max-Age=63071999; Path=/;  secure
Strict-Transport-Security: max-age=31536000; includeSubdomains
Via: 1.1 varnish
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
access-control-allow-origin: *
access-control-expose-headers: X-Moose
content-type: application/json; charset=UTF-8
x-moose: majestic
x-ratelimit-remaining: 0
x-ratelimit-reset: 178
x-ratelimit-used: 100
x-ua-compatible: IE=edge

{
    "error": 429,
    "message": "Too Many Requests"
}

So you're running into rate limiting. In your original example, this is probably due to querying multiple subreddits "at once". In my condensed version, I've reduced the wait period between pulls to achieve a similar effect.

The Reddit API rules say the following about rate limits:

As of July 1, 2023, we will enforce two different rate limits for those eligible for free access usage of our Data API. The limits are:

If you are using OAuth for authentication: 100 queries per minute (QPM) per OAuth client id
If you are not using OAuth for authentication: 10 QPM

QPM limits will be an average over a time window (currently 10 minutes) to support bursting requests.

Since your code does not use OAuth, you have 10 QPM, which the code violates -- so you're getting a HTTP 429 after the documented 10 minutes window.

You will need to switch to OAuth or use a longer delay between polls.

Unfortunately, neither you as a user of roux-stream nor roux-stream itself can do much else here, since roux does not expose any details about the HTTP interactions with the Reddit API. I've opened halcyonnouveau/roux#73 for discussing how to improve that.

As there is nothing else to do w.r.t. roux-stream here, I will close this ticket.

@torfsen torfsen closed this as completed Aug 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants