Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"data/sessions" folder growed up and used 100% inodes #6320

Closed
Perflyst opened this issue Mar 13, 2019 · 19 comments

Comments

@Perflyst
Copy link

commented Mar 13, 2019

  • Gitea version (or commit ref): 1.7.3
  • Git version: 2.11.0
  • Operating system: Debian 9
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist:

Description

Today I got a warning from my monitoring that the gitea machine has no more inodes left. Gitea crashed and you couldnt write on the disk anymore. I removed data/sessions/* which had incredible much inodes.

Is there any work around? Is this a bug? Why does it grow soo much?

@lunny

This comment has been minimized.

Copy link
Member

commented Mar 14, 2019

If too many users are visiting your gitea site, it will create many session files that maybe use many inodes. You can change your session provider to memory, redis, mysql and etc. Currently you could delete all the session files to let gitea work, but then everyone have to login gitea again.

@lunny lunny added the kind/question label Mar 14, 2019

@Perflyst

This comment has been minimized.

Copy link
Author

commented Mar 14, 2019

If I use memory than probably all users are logged out if I restart gitea.

It just filled up again 80% inodes. Why do you not delete the session files automatically if the user just visits the site without logging in?

@FloThinksPi

This comment has been minimized.

Copy link

commented Mar 15, 2019

@Perflyst @lunny We also see this problem. We have aprox 200 Users, but Gitea seems to create a session file for each request! The save sessions in file function is obviously broken in 1.7.4. We see around 100 to 1000 Files per second beeing created. Our System Crashed at 20GBs and 5 Million files in the sessions folder. This is due to Docker trying to relable the selinux labels of all the files (:Z option in docker-compose) which leads to the docker deamon to hang/crash.

Our Configuration was:

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER        = file

Because we thought maybe the defaults are wrong we tried:

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER        = file
GC_INTERVAL_TIME = 3600
SESSION_LIFE_TIME = 86400
COOKIE_NAME = i_like_gitea
COOKIE_SECURE = true
ENABLE_SET_COOKIE = true

But no change whatsoever.

@Perflyst We solved it by changing the method from file to memory, wich is only suitable on singe instance server. If you have multiple gitea servers working in LoadBalancing configuration you would need to setup a redis to work around it.
Everytime you clear the sessions folder all users will have to login again, so next time you can just change to memory where all users have to log in gain also and afterwards delete the sessions folder wich is not needed anymore now.

Interresting also for @inxonic

@Perflyst

This comment has been minimized.

Copy link
Author

commented Mar 15, 2019

Just as a side note, I do not use docker (if this is relevant).
I will change to memory but still this can't be right. It worked before without 1.7.3 / .4 these much session files.

@FloThinksPi

This comment has been minimized.

Copy link

commented Mar 15, 2019

Yours crashed because you reached your inode limit, mine crashed earlier because the docker deamon refused its work :), it is not relevant for the problem. 5 Million Session files would be OK with 5 Million Users Logging in all within a 1 Day timeframe. However in our case and user counts there should be around 100 Session files normally. Something is fishy with session handling when using files.

Last Commit in /sessions/file.go was 9de871a which Added following lines:

// Skip encoding if the data is empty
if len(s.data) == 0 {
return nil
}

But i cannot see how this could case such issues.

@FloThinksPi

This comment has been minimized.

Copy link

commented Mar 15, 2019

@techknowlogick as you seem familliar with the session handling and file method, could you have a look on this ?

@lunny

This comment has been minimized.

Copy link
Member

commented Mar 15, 2019

@Perflyst @FloThinksPi could you confirm that v1.7.2 work for you?

@FloThinksPi

This comment has been minimized.

Copy link

commented Mar 15, 2019

@lunny our last version was commit 20c54f8 . We updated straight from that "non release" version to 1.7.4. In 20c54f8 we did not see this issue, though we had other crashed so our instance might crashed before we could obseve the session issue. But i think we would have noticed that many files so i would say in 20c54f8 it was not present.

@Perflyst

This comment has been minimized.

Copy link
Author

commented Mar 15, 2019

I can confirm that my monitoring never informed me about 80% inodes usage before 1.7.3

@FloThinksPi

This comment has been minimized.

Copy link

commented Mar 18, 2019

@Perflyst @lunny today our instace ran out of memory at 12+GB(8GB+4GB SWAP). So the bug is also present when using memory unfortunally. Though it takes much more time to fill the 12GB RAM+SWAP in comparison to filling 12GB Disk Space when using file.

@techknowlogick

This comment has been minimized.

Copy link
Member

commented Mar 18, 2019

@FloThinksPi have you tried using redis as session provider?

@zeripath

This comment has been minimized.

Copy link
Contributor

commented Mar 19, 2019

@techknowlogick I think there is probably a real issue here. Drive by visitors should not be having server sessions created. This never used to happen - something is writing to these sessions and we need to stop it.

@Perflyst is there anything suggestive of the cause in the session files to work out what is adding this data?

@Perflyst

This comment has been minimized.

Copy link
Author

commented Mar 19, 2019

Well, this just kills my terminal

$ cat 1156652f319d4a45
#⎽├⎼☃┼±                                                                                                                                                                           _⎺┌d_┤☃d⎽├⎼☃┼±▮±☃├e▒@±☃├e▒:·/d▒├▒/⎽e⎽⎽☃⎺┼⎽/1/1

I noticed that the session folder also grows if there are no visitors. It seems like the sync from a mirror also fills the session folder.

I don't know if this is relevant, but I see a lot 2019/03/19 08:40:28 [...ules/context/repo.go:625 func1()] [E] GetCommitsCount: Unsupported cached value type: <nil> in the gitea.log file

@FloThinksPi

This comment has been minimized.

Copy link

commented Mar 19, 2019

@techknowlogick we only tried memory and file, which seem to behave the same.

I browsed a litte trough the last commits and found something interresting.

// Read returns raw session store by session ID.
func (p *FileProvider) Read(sid string) (_ RawStore, err error) {
filename := p.filepath(sid)
if err = os.MkdirAll(path.Dir(filename), 0700); err != nil {
return nil, err
}
p.lock.RLock()
defer p.lock.RUnlock()
var f *os.File
if com.IsFile(filename) {
f, err = os.OpenFile(filename, os.O_RDONLY, 0600)
} else {
f, err = os.Create(filename)
}
if err != nil {
return nil, err
}
defer f.Close()
if err = os.Chtimes(filename, time.Now(), time.Now()); err != nil {
return nil, err
}
var kv map[interface{}]interface{}
data, err := ioutil.ReadAll(f)
if err != nil {
return nil, err
}
if len(data) == 0 {
kv = make(map[interface{}]interface{})
} else {
kv, err = DecodeGob(data)
if err != nil {
return nil, err
}
}
return NewFileStore(p, sid, kv), nil
}

The Read function in managers providers does crate a file if none is found yet. So it does not only read, it will also write new sessions. Maybe one should split this functionality but nevermind for this issue.

So in:

9de871a

A validator function got intoduced. Now every call should include this validator function.
However in the same commit where the validator function was introduced, the call to the Read function changed from m.Read to m.provider.read:

image

So in my opinion this would bypass the managers read function and directly call the provides read function which does not validate the session further.

// Read returns raw session store by session ID.
func (m *Manager) Read(sid string) (RawStore, error) {
// Ensure we're trying to read a valid session ID
if _, err := m.validSessionID(sid); err != nil {
return nil, err
}
return m.provider.Read(sid)
}

I dont know if this could cause this but maybe invalid sessions are provided here to the read function and it creates them without further checks ? Just a guess here.

@lunny

This comment has been minimized.

Copy link
Member

commented Mar 19, 2019

I found v1.7.2 still has this problem, not only v1.7.4. When I refresh home page, /manifest.json will get a new session id every time.

[Macaron] 2019-03-19 20:04:22: Started GET /manifest.json for [::1]
2019/03/19 20:04:22 [D] Session ID: 1f35a522846589cc
2019/03/19 20:04:22 [D] CSRF Token: z440K8BT9R6lqI0k2b5TL3crBV06MTU1Mjk5NzA2MjMwNzAwNzAwMA==
2019/03/19 20:04:22 [D] Template: pwa/manifest_json
@Perflyst

This comment has been minimized.

Copy link
Author

commented Mar 19, 2019

@lunny

This comment has been minimized.

Copy link
Member

commented Mar 19, 2019

I think session on this route could be disabled since it's public and only for PWA.

@lunny

This comment has been minimized.

Copy link
Member

commented Mar 19, 2019

@Perflyst @FloThinksPi could you confirm that #6372 fixed your problem and I will send back port to v1.8 and v1.7.5 after it's merged on master.

@Perflyst

This comment has been minimized.

Copy link
Author

commented Mar 19, 2019

Can you share a binary? linux-amd64, if possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
5 participants
You can’t perform that action at this time.