Server crashes after 1-4 hours of workload #10950

Dwynr · 2020-11-22T04:56:44Z

Expected Behavior

Don't crash

Current Behavior

After 1-4 hours of Put/Get workload Minio crashes

Possible Solution

/

Steps to Reproduce (for bugs)

Start minio in standalone mode
Wait

Context

Maybe this is related to my setup having about 50.000 buckets with 50 million objects, but I'm not sure.

Your Environment

Version used (minio --version): minio version RELEASE.2020-11-19T23-48-16Z
Server setup and configuration: Standalone
Operating System and version (uname -a): Linux de1cluster1 4.15.0-124-generic Accessing dir as object should return ObjectNotFound #127-Ubuntu SMP Fri Nov 6 10:54:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I attached the log files.

log.txt

The text was updated successfully, but these errors were encountered:

harshavardhana · 2020-11-22T05:23:31Z

Looks like found the problem

fatal error: concurrent map iteration and map write

goroutine 12723471 [running]:
runtime.throw(0x1e1f2de, 0x26)
        runtime/panic.go:1116 +0x72 fp=0xc01cf76808 sp=0xc01cf767d8 pc=0x437952
runtime.mapiternext(0xc01cf76980)
        runtime/map.go:853 +0x554 fp=0xc01cf76888 sp=0xc01cf76808 pc=0x410db4
github.com/minio/minio/cmd.(*dataUsageCache).flatten(0xc01cf76ea0, 0x2d80645f418, 0x303873, 0x2b629, 0x248c5, 0x2b3985, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/minio/minio/cmd/data-usage-cache.go:302 +0xe9 fp=0xc01cf769f0 sp=0xc01cf76888 pc=0x15869a9
github.com/minio/minio/cmd.(*dataUsageCache).dui(0xc01cf76ea0, 0x1dd4216, 0x1, 0xc020cb2000, 0x18508, 0x1a666, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/minio/minio/cmd/data-usage-cache.go:213 +0xf8 fp=0xc01cf76b98 sp=0xc01cf769f0 pc=0x1585858
github.com/minio/minio/cmd.(*erasureServerSets).CrawlAndGetDataUsage.func2.1()
        github.com/minio/minio/cmd/erasure-server-sets.go:370 +0x35d fp=0xc01cf76d68 sp=0xc01cf76b98 pc=0x179b8fd
github.com/minio/minio/cmd.(*erasureServerSets).CrawlAndGetDataUsage.func2(0xc003e67100, 0xc00cb0ba00, 0xc01110cf60, 0xc020cb2000, 0x18508, 0x1a666, 0xc011107270, 0xc0181591a0, 0xc011107290,
 0xc000bd0020)
        github.com/minio/minio/cmd/erasure-server-sets.go:389 +0x18c fp=0xc01cf76f90 sp=0xc01cf76d68 pc=0x179bc2c
runtime.goexit()
        runtime/asm_amd64.s:1374 +0x1 fp=0xc01cf76f98 sp=0xc01cf76f90 pc=0x470701
created by github.com/minio/minio/cmd.(*erasureServerSets).CrawlAndGetDataUsage
        github.com/minio/minio/cmd/erasure-server-sets.go:348 +0x8af

Dwynr · 2020-11-22T05:54:10Z

Yep thats what I was looking at aswell. I tried disabling the usage crawler but I did not help (maybe I did it wrong). I upgraded the server binary today and ever since then it started crashing.

harshavardhana · 2020-11-22T05:56:34Z

Yep thats what I was looking at aswell. I tried disabling the usage crawler but I did not help (maybe I did it wrong). I upgraded the server binary today and ever since then it started crashing.

Disabling is simple export MINIO_DISK_USAGE_CRAWL_ENABLE=off @Dwynr

harshavardhana · 2020-11-22T05:57:03Z

Can you paste your command line?

Dwynr · 2020-11-22T05:59:20Z

Okay let me try disabling it then.

MINIO_BROWSER=off MINIO_STORAGE_CLASS_STANDARD=EC:2 MINIO_STORAGE_CLASS_RRS=EC:2 MINIO_ACCESS_KEY=accessKey MINIO_SECRET_KEY=secretKey minio server http://host/minio/data{1...8}

harshavardhana · 2020-11-22T06:05:59Z

MINIO_BROWSER=off MINIO_STORAGE_CLASS_STANDARD=EC:2 MINIO_STORAGE_CLASS_RRS=EC:2 MINIO_ACCESS_KEY=accessKey MINIO_SECRET_KEY=secretKey minio server http://host/minio/data{1...8}

why are you running a single host system in this manner?

http://host/minio/data{1...8}

you can simply run minio server /minio/data{1...8}

Dwynr · 2020-11-22T06:07:10Z

Yea I'm pretty new to Minio and I wanted to expand the cluster sooner or later. I figured that I need to run it this way then (and just add more hosts to the command line).

harshavardhana · 2020-11-22T06:08:49Z

Yea I'm pretty new to Minio and I wanted to expand the cluster sooner or later. I figured that I need to run it this way then (and just add more hosts to the command line).

You cannot expand in this manner @Dwynr - a standalone system cannot be converted to multiple nodes in this manner you will not get the distribution - i.e erasure-coded objects wouldn't be spread out.

If you want to expand you need to setup distributed mode first and then copy your content from your current cluster to new cluster using tools like mc mirror or aws s3 sync

harshavardhana · 2020-11-22T06:10:49Z

Can you try using minio server /minio/data{1...8} without disabling crawler and see if it fails? @Dwynr

Dwynr · 2020-11-22T06:16:45Z

Yea I'm pretty new to Minio and I wanted to expand the cluster sooner or later. I figured that I need to run it this way then (and just add more hosts to the command line).

You cannot expand in this manner @Dwynr - a standalone system cannot be converted to multiple nodes in this manner you will not get the distribution - i.e erasure-coded objects wouldn't be spread out.

If you want to expand you need to setup distributed mode first and then copy your content from your current cluster to new cluster using tools like mc mirror or aws s3 sync

Thanks for your help! I wanted to create new nodes for the cluster anyways. So are you saying I should start a new cluster on 2 nodes in distributed mode and then just mirror all data from the old one? And then I can expand it as needed?

Can you try using minio server /minio/data{1...8} without disabling crawler and see if it fails? @Dwynr

I will now. Will report back if it crashes again. Thanks for your help again. Much appreciated.

this is needed to avoid initializing notification peers that can lead to races in many sub-systems fixes minio#10950

Dwynr · 2020-11-23T16:38:41Z

So I the server ran fine for almost a day now with the crawler disabled and your command line suggestions, but now I get

`
runtime: failed to create new OS thread (have 10802 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

runtime stack:
runtime.throw(0x1dde293, 0x9)
runtime/panic.go:1116 +0x72
runtime.newosproc(0xc0476c1c00)
runtime/os_linux.go:161 +0x1c5
runtime.newm1(0xc0476c1c00)
runtime/proc.go:1837 +0xdd
runtime.newm(0x0, 0xc000061800, 0x2a31)
runtime/proc.go:1816 +0x9b
runtime.startm(0xc000061800, 0x3252600)
runtime/proc.go:1973 +0xc9
runtime.handoffp(0xc000061800)
runtime/proc.go:2001 +0x52
runtime.retake(0x52f20429996, 0xc000050000)
runtime/proc.go:4826 +0x168
runtime.sysmon()
runtime/proc.go:4734 +0x1b1
runtime.mstart1()
runtime/proc.go:1172 +0xc8
runtime.mstart()
runtime/proc.go:1137 +0x6e
`

I got open files limit and user processes limit set to more than enough on the machine.

core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127352 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 640000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

I didnt change anything (still the same workload).

) this is needed to avoid initializing notification peers that can lead to races in many sub-systems fixes #10950

harshavardhana · 2020-11-23T17:09:04Z

So I the server ran fine for almost a day now with the crawler disabled and your command line suggestions, but now I get

`
runtime: failed to create new OS thread (have 10802 already; errno=11)
runtime: may need to increase max user processes (ulimit -u)
fatal error: newosproc

runtime stack:
runtime.throw(0x1dde293, 0x9)
runtime/panic.go:1116 +0x72
runtime.newosproc(0xc0476c1c00)
runtime/os_linux.go:161 +0x1c5
runtime.newm1(0xc0476c1c00)
runtime/proc.go:1837 +0xdd
runtime.newm(0x0, 0xc000061800, 0x2a31)
runtime/proc.go:1816 +0x9b
runtime.startm(0xc000061800, 0x3252600)
runtime/proc.go:1973 +0xc9
runtime.handoffp(0xc000061800)
runtime/proc.go:2001 +0x52
runtime.retake(0x52f20429996, 0xc000050000)
runtime/proc.go:4826 +0x168
runtime.sysmon()
runtime/proc.go:4734 +0x1b1
runtime.mstart1()
runtime/proc.go:1172 +0xc8
runtime.mstart()
runtime/proc.go:1137 +0x6e
`

I got open files limit and user processes limit set to more than enough on the machine.

core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127352 max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 640000 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

I didnt change anything (still the same workload).

As the error message indicates you need to increase ulimit -u

Dwynr · 2020-11-23T17:11:16Z

Well the limit for processes is already at max user processes (-u) 640000. That should be enough no?

harshavardhana · 2020-11-23T18:26:15Z

Well the limit for processes is already at max user processes (-u) 640000. That should be enough no?

According to the error its not @Dwynr - it looks like you are co-hosting this with other applications which are eating away the number of processes on the system

Dwynr added community triage labels Nov 22, 2020

harshavardhana assigned klauspost Nov 22, 2020

harshavardhana added priority: high and removed triage labels Nov 22, 2020

harshavardhana added a commit to harshavardhana/minio that referenced this issue Nov 22, 2020

fix: all hosts local and port same should be local erasure setup

8c7dd8d

this is needed to avoid initializing notification peers that can lead to races in many sub-systems fixes minio#10950

harshavardhana removed the priority: high label Nov 22, 2020

harshavardhana self-assigned this Nov 22, 2020

harshavardhana mentioned this issue Nov 22, 2020

fix: all hosts local and port same should be local erasure setup #10951

Merged

6 tasks

harshavardhana added a commit to harshavardhana/minio that referenced this issue Nov 23, 2020

fix: all hosts local and port same should be local erasure setup

6f6377b

this is needed to avoid initializing notification peers that can lead to races in many sub-systems fixes minio#10950

harshavardhana closed this as completed in #10951 Nov 23, 2020

harshavardhana added a commit that referenced this issue Nov 23, 2020

fix: all hosts local and port same should be local erasure setup (#10951

734d07a

) this is needed to avoid initializing notification peers that can lead to races in many sub-systems fixes #10950

github-actions bot locked as resolved and limited conversation to collaborators Feb 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Server crashes after 1-4 hours of workload #10950

Server crashes after 1-4 hours of workload #10950

Dwynr commented Nov 22, 2020 •

edited

Loading

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020 •

edited

Loading

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020 •

edited

Loading

Dwynr commented Nov 23, 2020 •

edited

Loading

harshavardhana commented Nov 23, 2020

Dwynr commented Nov 23, 2020

harshavardhana commented Nov 23, 2020 •

edited

Loading

Server crashes after 1-4 hours of workload #10950

Server crashes after 1-4 hours of workload #10950

Comments

Dwynr commented Nov 22, 2020 • edited Loading

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020 • edited Loading

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

harshavardhana commented Nov 22, 2020

Dwynr commented Nov 22, 2020 • edited Loading

Dwynr commented Nov 23, 2020 • edited Loading

harshavardhana commented Nov 23, 2020

Dwynr commented Nov 23, 2020

harshavardhana commented Nov 23, 2020 • edited Loading

Dwynr commented Nov 22, 2020 •

edited

Loading

Dwynr commented Nov 22, 2020 •

edited

Loading

Dwynr commented Nov 22, 2020 •

edited

Loading

Dwynr commented Nov 23, 2020 •

edited

Loading

harshavardhana commented Nov 23, 2020 •

edited

Loading