User storage quota enforcement

[justinwb]

Need ability to do in-line enforcement of per user storage quotas.

[melvincarvalho]

So, this can happen at the OS level, at the node application level, or with a process sitting in the middle. A combination of 3 of these can happen too.

We should perhaps work on some generic helper scripts to analyze existing pods, in a way that gives a sys admin the stats they need, without compromising any user privacy.

[melvincarvalho]

Examples

• A script to find out the number of accounts on a pod.

• A script to find out the total, mean, median, min and max disk usage on a pod, without revealing the username

• A script to count the number of accounts above accounts above a certain quota

This could lead to setting sensible limits, based on the server capabilities, and leading to defaults that can be communicated to the user.

In a previous system, we had a little bar on the dashboard, showing your total disk allowance, and how full you are. That could work quite well.

[kjetilk]

Starting on this now, but since we are now just doing urgent things, my plan is to record a quota in the users database, and find something that works like UNIX du, and show the user a bar with their allowance, and then do something in the write operations to stop them from exceeding it.

[melvincarvalho]

And excellent start. Can we do this in RDF?

[csarven]

One way to generalise this issue would be to have server-centric resources that can include desired content.

[kjetilk]

Oh, yes, @csarven . But right now, we're not looking to generalize, that's for later, we're just fighting fires now :-)

[kjetilk]

Let me see, @melvincarvalho !

[melvincarvalho]

<#me>  solid : quotaMax    1000000 # bytes
<#me>  solid : balancerRemaining   500000 # bytes

Could be roughly what we need.

It's also close to how I've modeled payments, so we could at some point integrate payemnts and quotas, is why I ask ...

EDIT : our previous PHP servers had a working quota bar on the user dashboards, so that might provide some insights.

[kjetilk]

Ah, great!

But that should not be writeable to the user, so where do I store it? Somewhere in settings, with an ACL that grants only read access?

But how we go for MBs as the smallest unit of quota?

[melvincarvalho]

@kjetilk where is user specific stuff written?

Note : it doesnt have to be STORED in RDF, but it can still be modeled in RDF.

Yeah I guess we have a posix vocab already, I wonder if there's something reusable there like blocks. Getting down to the byte level would be cool, but how wide is the int that stores it?

[kjetilk]

Yeah, but I figured if is stored as RDF, I can do it right away, otherwise you'd need to file a feature request. :-)

So, the obvious place to store it in the current arch is in the users part of the key-value store and implement it in the UserAccount class, which is what I started doing. But it is certainly also an option to store in a file in settings along with prefs.ttl et al... I don't know what would be involved to do that, but I'll check it out. I hope you don't mind me taking the path of least resistance right now :-) That path will not be taken frequently, once we get head above water.

[melvincarvalho]

I hope you don't mind me taking the path of least resistance right now

Of course!

Any future unexpected reuse is a bonus.

I suggest writing out the design before diving in, would be useful.

So, the obvious place to store it in the current arch is in the users part of the key-value store and implement it in the UserAccount class, which is what I started doing

Yes, so we need an ephemeral value showing how much disk is left.

And we need either a server wide quota limit, or a per user quota limit, or a default than can be upped (say for different tiered usage)

[melvincarvalho]

Just sprung to mind. @megoth did a cool thing yesterday where he created a file that he could read, but not write to (in his own space). Perhaps this is the path to creating system files.

We have a file in /profile/ something like quota.ttl and set the acl to read only so only the system can write to it. Just an idea (thinking outside the box).

[melvincarvalho]

profile probably the wrong place for it`

maybe .system or .well-known/solid/

[kjetilk]

Yes, that was what I had in mind. But settings seems even better.

So, I found the posix-stat, which has size, but I would have preferred a datatype, e.g.

<#me> solid:nvmQuota "25"^^ex:megabyte .

[dmitrizagidulin]

created a file that he could read, but not write to (in his own space).

That seems.. problematic :) Are we really going to store files over which the user has no control, in their data space, DRM-style?

[dmitrizagidulin]

Hmm. On second thought, I guess we do need to store system-controlled data eventually. (To record the author, access logs, etc.) Never mind :)

[megoth]

Does it need to be stored in their data space though? We could manage a separate data space where system-controlled data are stored, couldn't we? Just so that we don't mix system-controller and user-controller files.

[kjetilk]

Given the review in #911, this now only relies on solid/vocab#35

[Ryuno-Ki]

Yeah I guess we have a posix vocab already, I wonder if there's something reusable there like blocks. Getting down to the byte level would be cool, but how wide is the int that stores it?

Something I've learned while putting Gentoo on a Raspberry Pi: There's aa difference between size (in bytes) and number of inodes (in an ext4 filesystem).

That is, du (iirc) can still report free space while du -i tells you that your disk is full. I can look up details if you need. Maybe another issue? :-)

[kjetilk]

That is, du (iirc) can still report free space while du -i tells you that your disk is full. I can look up details if you need.

That's not necessary. It uses https://www.npmjs.com/package/get-folder-size , we take what we get from there.