added login-based workaround to view NSFW content

[PrivacyDevel]

Based on, preferred over and resolves #825
Closes #829 (and closes #302)

[roryyamm]

How is this going to work in the end?

[PrivacyDevel]

If this pull request got accepted and people updated, they would have the choice to either

• leave everything as is and nothing would change
• create a Twitter account, login and copy the required information from the sent HTTP headers / cookie list of their browser into the nitter.conf file

[bbappserver]

@PrivacyDevel This is inconvenient to use when the values can change around if using it in a docker container.
Please change it to

cfg.get("Config", "cookieHeader",  getEnv("cookieHeader", "") )
cfg.get("Config", "xCsrfToken",  getEnv("xCsrfToken", "") )

[bbappserver]

When running docker containers through nice web frontends this makes it easy to change the files without needing to screw around with the twitter conf and getting the cookieHeader all properly escaped for the .conf, because you can just paste it into a text box and it handles the escaping.

[xbdmHQ]

How to find the ct0 and auth token?

Right click and click developer mode or press F12 and click the tab that says "Network". Then find your twitter account username, usually at the top and click "Cookies".

How to find the xCsrfToken?

Right click and click developer mode or press F12 and click the tab that says "Network". Then in the filter type, "api".

[somini]

Does this "fix" #919? For public instances this might be hard to implement, but for personal instances, this would be acceptable.

[jiatern]

Does this "fix" #919? For public instances this might be hard to implement, but for personal instances, this would be acceptable.

I just set up a private instance, it works well.

[Mendelrock]

Does this "fix" #919? For public instances this might be hard to implement, but for personal instances, this would be acceptable.

I just set up a private instance, it works well.

Would you mind sharing your ways?
I tried to set it up too, but so far, I still get route errors. Well, to be precise, I can still view individual tweets, but trying to display a user's profile leads to the usual
An error has occured in one of your routes.
Detail: jsony.nim(45) parseHook
asyncfutures.nim(389) read
asyncfutures.nim(389) read
asyncfutures.nim(389) read
asyncfutures.nim(389) read
asyncfutures.nim(389) read
asyncfutures.nim(389) read
Expected { but end reached. At offset: 0
I probably missed a step, but I can't seem to find which :'(

[jiatern]

Would you mind sharing your ways?

I am not sure what you missed, but these are my steps:

1. Clone PrivacyDevel's fork https://github.com/PrivacyDevel/nitter, I use the master branch
2. Install all dependencies as instructed (libpcre, libsass, redis, nimble)
3. Update configurations in nitter.conf, ensure ct0, auth_token and xCsrfToken are also updated
4. Build nitter by following the guidelines
5. Run redis, and then run nitter

I am able to access individual user profiles as well.

[fnxweb]

It's not working for me, but I think it may be more fundamental, as I've not built nitter before. For a start, the Nitter "front page" is stuffed, with over-sized images, suspect some CSS is AWOL.
Then if I try to view a user, I actually see 404s coming back from Twitter – is that a symptom of not having working ct0/auth_token/XCsrfToken values? I have double checked that what is in the Chrome dev. network data is what I have in the config file.

404 Not Found: --- url: https://api.twitter.com/graphql/somecodehere/UserByScreenName?variables=...etc

[fnxweb]

The CSS is probably unrelated to the main functionality not working, but if I fetch http://localhost:8080/css/style.css I actually get an HTML Nitter error not the CSS file.

[fnxweb]

CSS fixed, I had neglected the nimble scss (which by its name is plainly related!)

Still get 404 out of Twitter though :(

[PrivacyDevel]

CSS fixed, I had neglected the nimble scss (which by its name is plainly related!)

Still get 404 out of Twitter though :(

Does it work when you are using my fork instead?
Did you accidentally add spaces to those lines where there shouldn't be any?
Did you make any other big changes to the conf file?
What's the version string on the /about page?

[fnxweb]

I started from a git clone of nitter, created a new branch and pulled in your nsfw pull req. I've not yet tried a direct clone of your branch, but I will.

I copied my working nitter.conf and added the two new auth fields which I'll partially paste below. I'll do a more detailed diff of that and your updated example in case I missed something else. I do have your changes locally, and have rebuilt. Oh. But I added some extra debug to print the new fields and that's not coming out, either meaning my config is not being picked up, or it's not somehow rebuilt properly?

.. in [Config] section after the comment describing them
cookieHeader = "ct0=e5...ff; auth_token=bb...42"
xCsrfToken = "e5...ff"

That's interesting, the ct0 and Csrf are the same, I'd not noticed that. Expected? I'm sure I double checked them.

Version 2023.07.06-bb7e239

[fnxweb]

It is rebuilding OK, but my debug is bogus, possibly due to not understanding nim, so I can't verify what it's using, if anything, for the auth headers. Tried to add echos to proc fetch. Builds but nothing printed, although I can adjust the error output that I get.

[fnxweb]

OK, a raw build of your branch behaves differently, I now get the error occured in one of your routes that @Mendelrock reported above. And the debug still reports a 404 coming back from the API URL. I think that error may be a symptom of the 404, as Z's master now has a JSON parser/exception fix that you haven't yet got, I think (but I have in my other build as I merged nsfw onto his latest).

So I still have some issue somewhere.

[fnxweb]

Right, not sure what I was doing wrong, but I do get debug now, and upon the 404 I print my auth tokens and they are as I expect. But I can see zero evidence of them being used. The only time they are picked up is in the proc fetch (as above) and debug does not show this being called. I get errors from proc fetchRaw.

It is a bit more complex as well. If I search for the user (BungieHelp FWIW) I do get results. But if then click on a user name (that or any others), it comes back not found (well, a 404, which Z's master has a fix-ish for, so "not found" instead of the above routes error message).

I'm wondering: does your patch only add auth to the points where you found it was previously required (like search) and it is not present in the path that is used for a user view / feed (and so that now fails)?

[fnxweb]

Confirmed! This fixes user feeds:

diff --git a/src/apiutils.nim b/src/apiutils.nim
index bc4b498..35de0b6 100644
--- a/src/apiutils.nim
+++ b/src/apiutils.nim
@@ -122,6 +122,12 @@ proc fetch*(url: Uri; api: Api; additional_headers: HttpHeaders = newHttpHeaders
       raise rateLimitError()
 
 proc fetchRaw*(url: Uri; api: Api; additional_headers: HttpHeaders = newHttpHeaders()): Future[string] {.async.} =
+
+  if len(cfg.cookieHeader) != 0:
+      additional_headers.add("Cookie", cfg.cookieHeader)
+  if len(cfg.xCsrfToken) != 0:
+      additional_headers.add("x-csrf-token", cfg.xCsrfToken)
+
   fetchImpl(result, additional_headers):
     if not (result.startsWith('{') or result.startsWith('[')):
       echo resp.status, ": ", result, " --- url: ", url

[y-bonfire]

I tried this pr today. While RSS retrieval is functioning, it seems that the search functionality is not working properly.
1234b9b
The same situation?
#983 (comment)