Moving toward our next release

[sigaloid]

It is done. https://github.com/redlib-org/redlib

Hello all,

Regretfully I haven’t had the free time to keep up with the recent changes a few days ago. reddit’s rate limiting is successfully implemented. I want to get a new release out ASAP (this weekend hopefully) that will do a few things. I would be grateful if someone would be able to help with any of these changes.

• Modify my draft PR- use it as a base since it has some needed changes but tear out all of the oauth spoofing. Currently it auto generates an API token via spoofing the mobile app. We cannot distribute code that does this. Change it to instead have an optional config option to have a token. DO NOT return it in the info+config page! The change you should keep is the change of the base URL to oauth (if the config option is set) and the attaching of any relevant headers (not fake mobile ones). the whole oauth module should be thrown out.

• See exactly how the rate limits work. I know they said 10/min for “anonymous” “Oauth” requests (I take this to mean oauth.reddit.com requests without ANY auth). Does this work correctly? Is it better or worse than anonymous? Basically, does oauth.r have any different limits than www.r for the same endpoints (.json ones - not full oauth specced ones)?

• Also, they said 100/min for “authenticated free oauth requests” meaning to oauth.r but with a free token - can these be generated live? This is exactly what reveddit did in client side to check the “live” version of the app. Basically these tokens were free to generate with a single http request and had no validation. Is this still possible, or is there required to be an account attached now? To do this, take a look at my draft PR, look at the oauth.rs initialization routine and see what request it makes. Translate it to curl and make sure it still works (it should ;) ). (Alternatively in the reversing reddit api issue there was python code someone posted that did similar- see if it still works). Then strip out the mobile spoofed headers - does it still return a token? If so, we can plausibly use it to generate a token at launch. 100r/min might be more breathing room. If possible/safe to do, we could generate them dynamically based on demand and distribute requests across a few tokens.

• We neeeeeeed to rename. I can’t emphasize this enough. Doubly so, since if we make these changes, we’re implicitly agreeing to the API’s terms, which describe exactly what words we can use and can’t use to define our project. We need to replace all of the instances of “reddit” in the codebase, templates, etc.

• Also, what is going to happen when we agree to the API terms? This is going to have ramifications since before we were just web scraping but now the implicit API terms will change that.

• Merge the recent PR about the de-anonymizing headers. Critical priority - only reason I haven’t merged is that we need to get these other changes out with the next release - there’s no point in pushing another release when the behavior is borked the way it is for any popular instance.

All of these are top priority for the project and I want to get a release out soon.

To be clear I have been asked by operators whether this would require them creating a token (possibly linked to an account) to run an instance and that they likely wouldn’t run one if they had to. My goal is to avoid this if at all possible.

If your traffic is below the rate limits right now you already can. It may be possible that even if it’s higher, you can either 1. have the application generate one of these “anonymous tokens” on the fly without account cookies, logging in, or anything, or 2. specifying one of your own (which can come from anywhere - it doesn’t have to be one you make). I know several apps are given Reddit’s blessing to escape rate limits and someone will surely extract their token. We can’t redistribute it but we also can’t stop you from using it.

Sorry for the megapost. I’m going to try to make some progress on these in the coming days (me posting this is also lighting a fire under me to do exactly that) but if anyone’s looking to contribute it would be a huge help.

Truth be told I’m losing some hope for this open source front end genre of software, with all of the changes happening. The twitter API change threw an ugly wrench into a lot of my work and research, and this throws a medium ugly wrench into Libreddit, but I trust that it’ll work out fine (and I know this kind of thing is needed even more because of these changes).

Anyway, peace, and if you’re interested in picking up any of these (I think 2, 3, or 4 would be the most concrete) and have questions, let me know. I’ll keep my progress updated here.

PS: If you want to help but don’t want to do any of these tasks, link this issue in the bug reports that talk about rate limiting. We’re working on it folks! Also instance operators will have to upgrade once we stamp out a new release so there’ll be more waiting.

• PPS: Another easy task is that there’s a couple of outstanding requests to add instances in our instances repo. Turn them into PRs!!

[AyoungDukie]

For name at least, it may be worth considering the former fork's name ferritreader/ferrit

[0xEsky]

Than you so much for this. Seems a very sound and reasonable approach.

[CONIGUERO]

@sigaloid why not just spoof the mobile API? no need to agree to any terms. https://news.ycombinator.com/item?id=36086240

There's work being done under #818 too!

EDIT: Just saw item #1 on the checklist. Why can't you distribute the code that does that? It's nothing illegal and, as you said earlier, the project has not agreed to any API terms.

[sigaloid]

Yes, I was part of that effort to reverse engineer, but we can’t distribute code that emulates or spoofs an official client. It opens us up to legal concerns.

[CONIGUERO]

Yes, I was part of that effort to reverse engineer, but we can’t distribute code that emulates or spoofs an official client. It opens us up to legal concerns.

It does not. It's what clients for scrape-hostile sites like YouTube do. NewPipe, for example, impersonates an Android TV and web client to get the YT data and video streams. It's also what Nitter does (using the twitter web client API keys), which is a project very similar in both nature and spirit to this one.

[htsmi]

It may be time to consider spinning off of libreddit as a new project. It sounds like it needs to be renamed anyhow and will have to take a considerably different approach. Whatever that is will have to contend with a lot more hostility from the Reddit side, some people will be willing to go further than others in this regard. It would likely be good to have some separation from the current project, and those who have contributed to it, because they may not want to be associated with the approaches that are now required.

[sigaloid]

It does not.

Just because others do it, doesn’t mean it’s not grounds for a takedown or whatever legal action is necessary. What many seem to misunderstand about this kind of law is that even if these actions are okay to perform, that won’t stop an army of lawyers from making whatever threats necessary. This isn’t some unknown thing, this has happened to maintainers of similar software who have zero standing besides “you’re breaking TOS”, yet the legal threats were enough to force his hand into giving up on it. Youtube-dl, invidious, barinsta.. all who have received these threatening letters. I’m not willing to place my full legal name onto what very well may be an infringement. In another universe where my contributions weren’t in my name and this wasn’t the stance of the whole Libreddit team, things would be different. The biggest defense against a notice, is a lack of destination address.

I hope you don’t take this as me not supporting that kind of reverse engineering change (I literally wrote half the code for it). My opposition is primarily because it’s the stance of the whole team, and secondly, that I believe it will not be necessary. It’s not that I don’t want to be associated with these issues due to a moral obligation.

If it indeed becomes necessary, and we can’t get by with the changes outlined, Libreddit (the current team and repo) will shut down anyway, in which case I would be a supportive user of a fork that keeps it alive. I only ask that it wait until I have the time to confirm it. I’m juggling a lot of things right now but I intend to get to this worked on soon.

Why I don’t believe it will be necessary to fork: At the worst case, we will add a token setting, someone will pluck out the token off of another app and share it. This change on our end can be done in a lunch break if I cannot find another avenue.

[CONIGUERO]

It does not.

Just because others do it, doesn’t mean it’s not grounds for a takedown or whatever legal action is necessary. What many seem to misunderstand about this kind of law is that even if these actions are okay to perform, that won’t stop an army of lawyers from making whatever threats necessary. This isn’t some unknown thing, this has happened to maintainers of similar software who have zero standing besides “you’re breaking TOS”, yet the legal threats were enough to force his hand into giving up on it. Youtube-dl, invidious, barinsta.. all who have received these threatening letters. I’m not willing to place my full legal name onto what very well may be an infringement. In another universe where my contributions weren’t in my name and this wasn’t the stance of the whole Libreddit team, things would be different. The biggest defense against a notice, is a lack of destination address.

I hope you don’t take this as me not supporting that kind of reverse engineering change (I literally wrote half the code for it). My opposition is primarily because it’s the stance of the whole team, and secondly, that I believe it will not be necessary. It’s not that I don’t want to be associated with these issues due to a moral obligation.

If it indeed becomes necessary, and we can’t get by with the changes outlined, Libreddit (the current team and repo) will shut down anyway, in which case I would be a supportive user of a fork that keeps it alive. I only ask that it wait until I have the time to confirm it. I’m juggling a lot of things right now but I intend to get to this worked on soon.

Why I don’t believe it will be necessary to fork: At the worst case, we will add a token setting, someone will pluck out the token off of another app and share it. This change on our end can be done in a lunch break if I cannot find another avenue.

I appreciate the detailed response and respect the choice. However please don't implicate other projects that didn't do that. Except for Barinsta, all tbe other examples you gave refused to shut down even with a legal action threat, Invidious doing so very recently: iv-org/invidious#3872 (comment)

[htsmi]

Thanks for the clarification on your position and situation, that helps. I am encouraged that you still see a route forward on this. Thank you for all your hard work!

[avincent98144]

... yawn, next. On Jul 21, 2023, at 10:07 AM, mhd1msyb ***@***.***> wrote: @sigaloid There is no such this as "reverse engineering". That is the stupidest and most pathetic lie ever. Stop being disrespectful & insulting to Engineering. You know nothing about what Engineering entails. Engineering means designing, building and producing real mechanical systems, like turbojets and vehicular engines. It has nothing to do with clowns sitting behind a keyboard and decoding/programming/deciphering etc. Software dev is easy and for young kids and teens. And it has absolutely nothing to do with a serious and real subject like Engineering (Mechanical Engineering is the ONLY Engineering). Quit making false claims against engineering you. And stop being disrepectful to real engineers (Mechanical Engineering is the ONLY Engineering, PERIOD). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sigaloid]

Just to give an update, the team is still deliberating on how we’re going to move forward. I believe we can do so while keeping Libreddit easy to use and functional even under high traffic. I know the waiting period is frustrating!

[avincent98144]

looking forward to a sustainable solution. so far, no instance on any browser on any TOR, VPN or otherwise .. is working, eventually bringing back ’too may requests’. effing reddit…………..

…

On Jul 22, 2023, at 3:47 PM, Matthew Esposito ***@***.***> wrote: Just to give an update, the team is still deliberating on how we’re going to move forward. I believe we can do so while keeping Libreddit easy to use and functional even under high traffic. I know the waiting period is frustrating! — Reply to this email directly, view it on GitHub <#836 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRHFD7LBCLQHEVMTZ4QYODXRRJ6LANCNFSM6AAAAAA2ILEPZE>. You are receiving this because you commented.