Expanse data is not beeing updated

[Z0pyrus]

After some time I've run into an issue where the data is not getting updated anymore: last updated: 15h ago. I've imported my Reddit data with csv files which lead to an overall saved count of 4410. Also, my pihole fetched about 38000 requests to oauth.reddit.com.
In the console I got this error while it's trying to update the data:

RequestError: Error: ETIMEDOUT
    at new RequestError (/backend/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/backend/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/backend/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/backend/node_modules/request/request.js:185:22)
    at Request.emit (node:events:513:28)
    at Timeout.<anonymous> (/backend/node_modules/request/request.js:851:16)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7) {
  cause: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
  error: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
�      
  options: {
    gzip: true,
    json: true,
    headers: { 'user-agent': 'web:expanse:v=1.0.0 (hosted by u/myusername)' },
    baseUrl: 'https://oauth.reddit.com',
�      
{
      raw_json: 1,
      count: 9999,
      limit: 9007199254740991,
      before: 't3_xazslb'
    },
    auth: { bearer: 'hidden for privacy' },
    resolveWithFullResponse: true,
    timeout: 30000,
    transform: [Function: transform],
    uri: 'user/myusername/upvoted',
    method: 'GET',
    callback: [Function: RP$callback],
    simple: true,
    transform2xxOnly: false
  },
  response: undefined
}

At the moment I try to recreate the error:

I've imported csv files to it (I think all of the possible ones). After that it didn't updated anymore.

[Z0pyrus]

After trying to reproduce it, I get the error after logging in as well. Am I maybe banned for api calls for reddit?

While Logging in the first time it is stuck at some point which can be solved by reloading the page.
There is also a new error in the console:

Error: user (myusername) dne
    at Module.get (file:///backend/model/user.mjs:572:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async file:///backend/controller/server.mjs:60:13
Error: user (myusername) dne
    at Module.get (file:///backend/model/user.mjs:572:9)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async file:///backend/controller/server.mjs:60:13
destroyed session (myusername)
socket (ie7Gwvh4x1yOSbDBAAAD) connected
update all completed
update all completed
checking if user (myusername) exists
new user (myusername)
saved user (myusername)
socket (zRjQDEdhjnpt_h1-AAAF) connected
getting user (myusername)
deserialized user (myusername)
getting user (myusername)
updating user (myusername)
getting user (myusername)
update all completed
getting user (myusername)
update all completed
RequestError: Error: ETIMEDOUT
    at new RequestError (/backend/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/backend/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/backend/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/backend/node_modules/request/request.js:185:22)
    at Request.emit (node:events:513:28)
    at Timeout.<anonymous> (/backend/node_modules/request/request.js:851:16)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7) {
  cause: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
  error: Error: ETIMEDOUT
      at Timeout.<anonymous> (/backend/node_modules/request/request.js:848:19)
      at listOnTimeout (node:internal/timers:559:17)
      at processTimers (node:internal/timers:502:7) {
    code: 'ETIMEDOUT',
    connect: true
  },
�

While Importing the data I also get the ETIMEDOUT error some times.

[Z0pyrus]

I think limiting the calls my help because in the Reddit API docs is written that: “Clients connecting via OAuth2 may make up to 60 requests per minute”. So maybe I'm "banned".

During my first install I scrolled to the bottom of my saved posts. Maybe this could have triggered a ban if each entry was loaded somehow but this is very unlikely.

[aplotor]

hmm ive never run into this while testing. i just tested again with an import of over 20k items and its still as expected

from that error message i cant tell what caused it, other than sometimes reddit api is unreliable and you get undefined response, but that should not crash the update cycle. is there anything else specific you did before this happened? also, the request timeout error might possibly be on your device's end?

ill leave an instance up for now and see if it runs into this

[aplotor]

I've imported csv files to it (I think all of the possible ones). After that it didn't updated anymore.

i tested this and had no issue. so i cant really debug it. i guess we can wait for more ppl to use expanse and see if they get this too. but since it runs in docker it should be the same for everyone, so i think this might be a network issue on your device's end? are you sure your box didnt get disconnected from the internet as the request is being timed out?

[aplotor]

While Logging in the first time it is stuck at some point which can be solved by reloading the page. There is also a new error in the console

the dne error and 401 is intentional, if you deleted the db. this happens bc the cookie has a user session but the db doesnt have the user anymore. it is intentional and from expanse, not from reddit

[Z0pyrus]

I've tested it about 5 times now, and I always get the same error even during the initial run of expanse. I also changed my external IP address.
How long should be the session secret and encryption key be? Could be extreme long keys cause this? (128)

But like I said, I have a really "special" setup (raspberrypi 4, 8gb, docker in rootless mode on an SSD)

the dne error and 401 is intentional, if you deleted the db. this happens bc the cookie has a user session but the db doesnt have the user anymore. it is intentional and from expanse, not from reddit

Ok, I think I've got this error after deleting my cookies.

But are 40.000+ requests to reddit normal? (~24h)

[aplotor]

I think limiting the calls my help because in the Reddit API docs is written that: “Clients connecting via OAuth2 may make up to 60 requests per minute”. So maybe I'm "banned".

no, i do not think you are banned from the api. all requests from expanse are made from an reddit api wrapper (snoowrap) which has built-in ratelimit protection. ive also coded it so that the ratelimit should not be attempted to be reached regardless, and this has been battle-tested with eternity for around a year already

During my first install I scrolled to the bottom of my saved posts. Maybe this could have triggered a ban if each entry was loaded somehow but this is very unlikely.

this also could not have triggered a ban since scrolling your categories on expanse only retrieves items from your expanse db, it doesnt request them from reddit at all

[aplotor]

How long should be the session secret and encryption key be? Could be extreme long keys cause this? (128)

no, i dont think that could cause this, but it is unnecessary to be that long though. those are not passwords, those are used to compute hashes

But are 40.000+ requests to reddit normal? (~24h)

this is possible, especially if you had a lot of items to import. but the ratelimit should not be exceeded like i said due to the ratelimit protection as well as how i implemented the import process (imports are split up into intervals to ensure it does not attempt to request past the ratelimit). and 40,000 requests being split up over 24 hours does not exceed the ratelimit of 60req/min

once they are done importing, there should be less requests too. and if you have multiple users, each user has their own ratelimit

e: it's not really normal actually, but i think snoowrap retries the failed requests, on the docs it says "snoowrap will retry its request a few times if reddit returns an error", not sure if it also does for failed requests in general, like if they fail due to your network setup. so yea possible i guess, but still within ratelimit

so yes, right now i think it has something to do with your setup specifically. but we can see if other people run into this, not much else i can do now

[Z0pyrus]

each user has their own ratelimit

Also if the users are in the same Home Network? (This is just a general question)

[aplotor]

yep

[Z0pyrus]

I maybe found the culprit: It could be my reverse proxy (NGINX Proxy Manager). I will try to use the up only. (this helped me)
Update: Still the same error.

[aplotor]

it seems that both you and @rickcecil in #2 are using nginx proxy manager so i think it has something to do with that

for reference i am also using nginx (but with a manual setup, not nginx proxy manager) as a reverse proxy for eternity, and it does work fine

[Z0pyrus]

Maybe it has something to do with my pihole. I'm testing expanse on wsl right now, and I get another error which is similar. Maybe it's my dns (unbound) I will investigate more in this tomorrow.

[aplotor]

see my edit above. btw what are you using to check outgoing requests? and are you able to confirm if they are all/mostly failed requests? and the cause of failure?

[Z0pyrus]

I only see the failure in the logs. I'm also running a pihole where I can see how many requests were made to Reddit. But none of them failed.

After testing expanse on my Laptop on different networks I think I know why some requests are failing: It might be my pihole + unbound setup. But I don't know why this is happening.

After more invastigation I highly think it is my Unbound DNS causing this error.

The easiest solution bmight be to just set 9.9.9.9 (or any other DNS server) as the primery dns server for this container.

Another error I encounterd during this research was that when I imported "post votes" I got an 404 error serveral times.

[aplotor]

ok. and ive had an instance on wsl2 running for over 24h at this point with over 50k items imported, everything is fine. pretty certain this is not a problem with expanse so im just gonna close this now. if you fully figure it out feel free to comment after though, it might help others with a similar setup/config as you

404 is not found, so prob those items were deleted from reddit

[Keagel]

I'm not sure if it's related to this issue as there are no errors in the logs, but my content isn't being updated either. Here's an excerpt of the logs:

last Thursday at 9:49:37 PM updating user (Keagel)
last Thursday at 9:49:39 PM updated user (Keagel)
last Friday at 11:40:53 AM backed up db to file (23-09-2022_09꞉40꞉53꞉AM_UTC.sql)
last Saturday at 11:40:53 AM backed up db to file (24-09-2022_09꞉40꞉53꞉AM_UTC.sql)
yesterday at 11:40:53 AM backed up db to file (25-09-2022_09꞉40꞉53꞉AM_UTC.sql)
today at 11:27:37 AM socket (1W4pcrj0y8-iKP_OAACZ) connected
today at 11:27:37 AM getting user (Keagel)
today at 11:27:37 AM deserialized user (Keagel)

Restarting the container forces the updates though.

[aplotor]

this issue seemed like it was not a problem in expanse itself, but rather had to do with the end-user's network configs. if you think your issue is with expanse, please open a new issue for it