Heroku approaching row limit

[Dza-byte]

A day or two after deploying to Heroku using the deploy button, I have begun getting alerts that the database DATABASE_URL is approaching its allocated storage capacity. The number of rows continues to grow even though no data has been entered. I am not familiar enough with Heroku to have been able to find a solution on my own.

[cdubz]

That is… odd. I’ll do a test deploy to see if I can replicate. If you haven’t put any private data in, are you able to export data clips to see what exactly is using up rows?

[Dza-byte]

I'm willing to, however I don't know how to write the necessary SQL query to get the information I would need, let alone understand it. (I'm in way over my head here). Thanks for taking the time to look though!

[cdubz]

If you able to use the CLI tool it looks like you can use it to get an export. See https://devcenter.heroku.com/articles/heroku-postgres-import-export#export

[Dza-byte]

Okay, got the CLI going. Is it the dump file we need? It wont let me upload it here unfortunately.
https://drive.google.com/drive/folders/1rjwbNgz62QF4pw3IeieUBmJ-JsWaD6nt?usp=sharing
This is the dump file I get running "heroku pg:backups:capture"

[cdubz]

Bizarre... there are 17k+ sessions established according to the database. There should only ever be one session per user and browser. E.g. for the test instance I started when you posted this issue there are two sessions because I have logged in to it from two browsers.

Anyway, you can try to run the following command from Heroku CLI to clear out the sessions:

heroku run python manage.py clearsessions

If that doesn't work, I'd recommend just killing the instance and starting fresh.

I've never seen this before and I'm not quite sure what could cause it... please let me know if you are able to resolve or reproduce the issue.

[Dza-byte]

Thanks for your help! I will try both of those things and report back in a few days.

[mnoah66]

Hi @cdubz,
I am having a similar issue with Heroku where django_session fills up with a lot of session_keys. I am making API calls to my Heroku instance every 15 seconds or so (from Home Assistant).

I ran python manage.py clearsessions and truncated that table with pgAdmin. It has already filled up with 2,000+ rows in about an hour afterwards.

[cdubz]

Damn was really hoping that was a one-off freak accident thing (: We'll have to do some more digging on this, any help would be appreciated. I'm curious if this some (known or unknown) Django+Heroku or Django+Postgres bug.

@mnoah66 your screenshot is tiny but those timestamps look like the sessions are being created multiple times per second so this doesn't seem like API activity. Anything else you can tell me about your deployment? Any other settings or environment changes in Heroku outside of the regular button-click deploy? And maybe what browser + OS you're using to access Baby Buddy? Not quite sure what could be relevant here.

[Dza-byte]

I initially had the home assistant addon as well. I think the problem may lie with the way it's calling the api. Since removing it things have been fine, the home assistant link was not essential for me

[cdubz]

Ah good to know -- thanks @Dza-byte! That may help me figure out some way to reproduce...

[mnoah66]

Anything else you can tell me about your deployment? Any other settings or environment changes in Heroku outside of the regular button-click deploy?

Nothing was modified. Just the click-to-deploy button.

And maybe what browser + OS you're using to access Baby Buddy? Not quite sure what could be relevant here.

No browser, just via the baby_buddy_homeassistant add-on. It looks like the relevant code is in sensor.py. I occsaionally access my instance on Chrome(92.0.4515.107 )/Windows10, and Safari on iOS.

Still kind of new to all of this. Maybe the add-on is programmatically creating a new session for each call? I am not sure I can help much but I will try!

[cdubz]

Ok, cool. This is at least some lead (: I've got lots of pis lying around so I'll need to dust one off and put HA on it for testing. My understanding is that API calls really should not be creating sessions because they use token authentication but maybe something is misconfigured somewhere...

FYI @jcgoette if you have any thoughts/input on this thread.

[mnoah66]

Thanks @cdubz. I removed the Home Assistant add-on by @jcgoette and it is no longer creating a ton of session_keys.

[jcgoette]

Thanks for pinging me on this.

I don't know anything about Heroku, but I'm guessing the issue may be in this portion of my code.

This was one of my first "real" projects, so probably some bad code created here.

I'm thinking moving the session call into the init will leave it open and not generate new ones per sensor update. Will play around with tonight (time permitting).

[jcgoette]

I created jcgoette/baby_buddy_homeassistant#29, but since the reporters are here...

@Dza-byte or @mnoah66, would you be willing to reactivate and update baby_buddy_homeassistant to v1.3.4 and see if that solves this issue?

[Dza-byte]

Updated and rebooted. I'll post back if the issue comes back in the next day or two

[mnoah66]

Some improvement but in the last hour or so it has created ~400 session_keys (multiple w/in a second):

Also, looking at my Heroku logs. Looks like the same calls are being duplicated. I only have changes and feedings in my sensor configuration. It looks like it is calling the API 3x (with some redirects d/t missing trailing slash?). 3 is the ID of my child...I wonder if there is a for loop issue? Let me try adding another test child to see..

Let me know if you want to move this over to your repo, @jcgoette

[mnoah66]

After adding a test child (IDs of 3 and 4) the sequence of APIs are now being called 4x.

[jcgoette]

@cdubz, do you know if its possible/how to access the django_session table while using Django's development server (i.e. gulp)? Or do I need to try and spin up a Heroku instance?

@mnoah66, re: redirects, I opened jcgoette/baby_buddy_homeassistant#31.

Reducing API calls is also on my todo list: jcgoette/baby_buddy_homeassistant#26.

I am fine with moving this discussion over to jcgoette/baby_buddy_homeassistant#29, as I have a sense it is something on my end.

[cdubz]

@jcgoette you should definitely be able to see a django_sessions table in any context, including running the dev server with gulp. Just use whatever database interface you want for the storage engine you’re using.

[jcgoette]

@cdubz, thanks! Feel free to close this issue for now. If I somehow find it's an issue with babybuddy itself, I will ping you in Gitter.

[cdubz]

Thanks all for your efforts looking in to this!

This issue will be perused further in jcgoette/baby_buddy_homeassistant#29.

[cdubz]

@jcgoette @Dza-byte @mnoah66 turns out this was an issue with Baby Buddy's custom middleware! Fixed in f292698 and will make a new release later today.

[mnoah66]

Fantastic, thanks!

@cdubz Do you know if Heroku will pull the release automatically from this repo? Or do I need to re-deploy with the Heroku button (after the new release, ofc)? I would prefer to run locally on a Pi but couldn't get it figured out as I am not too familiar with Docker yet.

[cdubz]

@mnoah66 you'll have to redploy. I'm not 100% clear on how that works though... for the Herko-hosted demo I have Heroku's CLI and the instance added as a remote so I can just git push heroku from my local repo. But I wonder if you can get the same result without to checkout the code and go through the Heroku CLI and remote setup...

[mnoah66]

@cdubz tried the button and it just wanted to create a brand new app in Heroku. So went with the CLI and worked like a charm. Thanks again