Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fastdfs 经常CPU 100% #608

Closed
zhangjijian111 opened this issue Dec 20, 2022 · 33 comments
Closed

fastdfs 经常CPU 100% #608

zhangjijian111 opened this issue Dec 20, 2022 · 33 comments

Comments

@zhangjijian111
Copy link

store_path0下面文件夹中会出现
.cp5.tmp这样的文件

image
这时storage就会消耗CPU
@happyfish100
Copy link
Owner

FDFS用的什么版本？

@zhangjijian111
Copy link
Author

image
版本是6.9.1

@happyfish100
Copy link
Owner

CPU跑上去会持续多长时间呢？

@zhangjijian111
Copy link
Author

一直持续 出现这个的原因是 我两台服务器 我原来的版本是5.0.5 升级到6.9.1 tracker和 storage的配置所用的参数是原来的 其中一台服务器就会出现这个问题 另一台没有出现过

@happyfish100
Copy link
Owner

配置文件这块不存在兼容性问题，很奇怪会出现这样的现象。

@zhangjijian111
Copy link
Author

[root@YNET ~]#/usr/bin/fdfs_monitor /etc/fdfs/storage_datacenter_group1.conf list
[2022-12-21 16:18:18] DEBUG - base_path=/opt/data/FastDFS_Center/group1/fastdfs_storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=2, server_index=0

tracker server is 192.168.0.4:22122

group count: 4

Group 1:
group name = group1
disk total space = 10,737,418,240 MB
disk free space = 10,736,624,605 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 19092
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0

    Storage 1:
            id = 192.168.0.4
            ip_addr = 192.168.0.4  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2022-05-01 14:17:49
            up time = 2022-12-17 22:30:26
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23000
            storage_http_port = 19092
            current_write_path = 0
            source storage id = 
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 14735
            success_upload_count = 14735
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 4885
            success_download_count = 4885
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 12614958169
            success_upload_bytes = 12614958169
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 60056651381
            success_download_bytes = 60056651381
            total_sync_in_bytes = 11192080232
            success_sync_in_bytes = 11192080232
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 32922
            success_file_open_count = 32922
            total_file_read_count = 232634
            success_file_read_count = 232634
            total_file_write_count = 108001
            success_file_write_count = 108001
            last_heart_beat_time = 2022-12-21 16:17:58
            last_source_update = 2022-12-21 16:12:46
            last_sync_update = 2022-12-21 16:13:01
            last_synced_timestamp = 2022-12-21 16:12:55 (0s delay)
    Storage 2:
            id = 192.168.4.81
            ip_addr = 192.168.4.81  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2022-12-12 20:09:22
            up time = 2022-12-20 18:09:01
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23000
            storage_http_port = 19092
            current_write_path = 0
            source storage id = 192.168.0.4
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 104
            success_upload_count = 104
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 97
            success_download_count = 97
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 108065898
            success_upload_bytes = 108065898
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 3292869202
            success_download_bytes = 3292869202
            total_sync_in_bytes = 116468131
            success_sync_in_bytes = 116468131
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 314
            success_file_open_count = 314
            total_file_read_count = 13217
            success_file_read_count = 13217
            total_file_write_count = 1115
            success_file_write_count = 1115
            last_heart_beat_time = 2022-12-21 16:18:02
            last_source_update = 2022-12-21 16:12:55
            last_sync_update = 2022-12-21 10:18:36
            last_synced_timestamp = 2022-12-21 10:18:35 (05h:54m:11s delay)

Group 2:
group name = group2
disk total space = 10,737,418,240 MB
disk free space = 10,736,624,605 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23030
storage HTTP port = 19090
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

    Storage 1:
            id = 192.168.0.4
            ip_addr = 192.168.0.4  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2022-05-01 14:17:57
            up time = 2022-12-17 22:30:30
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23030
            storage_http_port = 19090
            current_write_path = 0
            source storage id = 
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 14653
            success_upload_count = 14653
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 5116
            success_download_count = 5116
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 12671954627
            success_upload_bytes = 12671954627
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 69248473789
            success_download_bytes = 69248473789
            total_sync_in_bytes = 10943442021
            success_sync_in_bytes = 10914241972
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 32974
            success_file_open_count = 32974
            total_file_read_count = 267728
            success_file_read_count = 267728
            total_file_write_count = 106957
            success_file_write_count = 106957
            last_heart_beat_time = 2022-12-21 16:18:04
            last_source_update = 2022-12-21 15:07:03
            last_sync_update = 2022-12-21 15:58:29
            last_synced_timestamp = 2022-12-21 15:58:20 (0s delay)
    Storage 2:
            id = 192.168.4.81
            ip_addr = 192.168.4.81  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2019-11-07 15:29:12
            up time = 2022-12-20 18:11:19
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23030
            storage_http_port = 19090
            current_write_path = 0
            source storage id = 
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 94
            success_upload_count = 94
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 98
            success_download_count = 98
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 88502190
            success_upload_bytes = 88502190
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 2851654571
            success_download_bytes = 2851654571
            total_sync_in_bytes = 185978509
            success_sync_in_bytes = 185978509
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 300
            success_file_open_count = 300
            total_file_read_count = 11301
            success_file_read_count = 11301
            total_file_write_count = 1200
            success_file_write_count = 1200
            last_heart_beat_time = 2022-12-21 16:17:50
            last_source_update = 2022-12-21 15:58:20
            last_sync_update = 2022-12-21 15:07:11
            last_synced_timestamp = 2022-12-21 15:07:04 (0s delay)

Group 3:
group name = group3
disk total space = 10,737,418,240 MB
disk free space = 10,736,624,605 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23060
storage HTTP port = 19096
store path count = 1
subdir count per path = 256
current write server index = 1
current trunk file id = 0

    Storage 1:
            id = 192.168.0.4
            ip_addr = 192.168.0.4  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2022-04-30 14:52:28
            up time = 2022-12-17 22:30:35
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23060
            storage_http_port = 19096
            current_write_path = 0
            source storage id = 
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 44476
            success_upload_count = 44476
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 78
            success_download_count = 78
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 58400938559
            success_upload_bytes = 58400938559
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 342420480
            success_download_bytes = 342420480
            total_sync_in_bytes = 47579791313
            success_sync_in_bytes = 47577891571
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 80183
            success_file_open_count = 80183
            total_file_read_count = 1342
            success_file_read_count = 1342
            total_file_write_count = 448962
            success_file_write_count = 448962
            last_heart_beat_time = 2022-12-21 16:18:09
            last_source_update = 2022-12-21 16:11:12
            last_sync_update = 2022-12-21 16:11:32
            last_synced_timestamp = 2022-12-21 16:11:26 (0s delay)
    Storage 2:
            id = 192.168.4.81
            ip_addr = 192.168.4.81  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2020-02-10 15:58:56
            up time = 2022-12-20 18:21:23
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23060
            storage_http_port = 19096
            current_write_path = 0
            source storage id = 
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 402
            success_upload_count = 402
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 0
            success_download_count = 0
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 502748654
            success_upload_bytes = 502748654
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 0
            success_download_bytes = 0
            total_sync_in_bytes = 704145160
            success_sync_in_bytes = 704145160
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 941
            success_file_open_count = 941
            total_file_read_count = 0
            success_file_read_count = 0
            total_file_write_count = 5298
            success_file_write_count = 5298
            last_heart_beat_time = 2022-12-21 16:17:53
            last_source_update = 2022-12-21 16:11:26
            last_sync_update = 2022-12-21 16:11:15
            last_synced_timestamp = 2022-12-21 16:11:12 (0s delay)

Group 4:
group name = group4
disk total space = 10,737,418,240 MB
disk free space = 10,736,624,605 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23080
storage HTTP port = 19098
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

    Storage 1:
            id = 192.168.0.4
            ip_addr = 192.168.0.4  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2022-05-01 14:18:04
            up time = 2022-12-17 22:30:39
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23080
            storage_http_port = 19098
            current_write_path = 0
            source storage id = 
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 44885
            success_upload_count = 44885
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 84
            success_download_count = 84
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 58959988344
            success_upload_bytes = 58959988344
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 367299072
            success_download_bytes = 367299072
            total_sync_in_bytes = 47197198069
            success_sync_in_bytes = 47197198069
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 80147
            success_file_open_count = 80147
            total_file_read_count = 1444
            success_file_read_count = 1444
            total_file_write_count = 449588
            success_file_write_count = 449588
            last_heart_beat_time = 2022-12-21 16:18:13
            last_source_update = 2022-12-21 15:43:13
            last_sync_update = 2022-12-21 16:05:25
            last_synced_timestamp = 2022-12-21 16:05:22 (0s delay)
    Storage 2:
            id = 192.168.4.81
            ip_addr = 192.168.4.81  ACTIVE
            http domain = 
            version = 6.9.1
            join time = 2022-05-01 14:21:38
            up time = 2022-12-20 18:21:46
            total storage = 10,737,418,240 MB
            free storage = 10,736,624,605 MB
            upload priority = 10
            store_path_count = 1
            subdir_count_per_path = 256
            storage_port = 23080
            storage_http_port = 19098
            current_write_path = 0
            source storage id = 
            if_trunk_server = 0
            connection.alloc_count = 256
            connection.current_count = 0
            connection.max_count = 0
            total_upload_count = 437
            success_upload_count = 437
            total_append_count = 0
            success_append_count = 0
            total_modify_count = 0
            success_modify_count = 0
            total_truncate_count = 0
            success_truncate_count = 0
            total_set_meta_count = 0
            success_set_meta_count = 0
            total_delete_count = 0
            success_delete_count = 0
            total_download_count = 0
            success_download_count = 0
            total_get_meta_count = 0
            success_get_meta_count = 0
            total_create_link_count = 0
            success_create_link_count = 0
            total_delete_link_count = 0
            success_delete_link_count = 0
            total_upload_bytes = 540677304
            success_upload_bytes = 540677304
            total_append_bytes = 0
            success_append_bytes = 0
            total_modify_bytes = 0
            success_modify_bytes = 0
            stotal_download_bytes = 0
            success_download_bytes = 0
            total_sync_in_bytes = 575171522
            success_sync_in_bytes = 575171522
            total_sync_out_bytes = 0
            success_sync_out_bytes = 0
            total_file_open_count = 951
            success_file_open_count = 951
            total_file_read_count = 0
            success_file_read_count = 0
            total_file_write_count = 4958
            success_file_write_count = 4958
            last_heart_beat_time = 2022-12-21 16:17:50
            last_source_update = 2022-12-21 16:05:21
            last_sync_update = 2022-12-21 15:43:18
            last_synced_timestamp = 2022-12-21 15:43:14 (0s delay)

我看现在都已经同步完成了 但是现在group1节点还是占用CPU

image

@happyfish100
Copy link
Owner

可以用 gdb attach上去打印一下调用堆栈信息吗？
可以在有问题的storage server上执行如下命令行：
echo 'thread apply all bt' | gdb attach -p $(pidof fdfs_storaged)

然后把输出的堆栈信息贴一下，方便我们排查问题，谢谢！

@zhangjijian111
Copy link
Author

[root@YNET ~]#echo 'thread apply all bt' | gdb attach -p $(pidof fdfs_storaged)
Excess command line arguments ignored. (24508 ...)
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
attach: 没有那个文件或目录.
Attaching to process 32658
Reading symbols from /usr/bin/fdfs_storaged...done.
Reading symbols from /lib64/libpthread.so.0...done.
[New LWP 32692]
[New LWP 32691]
[New LWP 32690]
[New LWP 32689]
[New LWP 32688]
[New LWP 32687]
[New LWP 32686]
[New LWP 32685]
[New LWP 32684]
[New LWP 32683]
[New LWP 32682]
[New LWP 32681]
[New LWP 32680]
[New LWP 32679]
[New LWP 32678]
[New LWP 32677]
[New LWP 32676]
[New LWP 32675]
[New LWP 32674]
[New LWP 32673]
[New LWP 32672]
[New LWP 32671]
[New LWP 32670]
[New LWP 32669]
[New LWP 32668]
[New LWP 32667]
[New LWP 32666]
[New LWP 32665]
[New LWP 32664]
[New LWP 32663]
[New LWP 32662]
[New LWP 32661]
[New LWP 32660]
[New LWP 32659]
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib/libfastcommon.so...done.
Loaded symbols for /usr/lib/libfastcommon.so
Reading symbols from /usr/lib64/libserverframe.so...done.
Loaded symbols for /usr/lib64/libserverframe.so
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcurl.so.4
Reading symbols from /lib64/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib64/libidn.so.11
Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /lib64/librt.so.1...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssh2.so.1
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
0x00007f9c149f591d in accept () at ../sysdeps/unix/syscall-template.S:81
81 ../sysdeps/unix/syscall-template.S: 没有那个文件或目录.
in ../sysdeps/unix/syscall-template.S
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-65.el6.x86_64 libcom_err-1.41.12-24.el6.x86_64 libcurl-7.19.7-54.el6_10.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-7.el6.x86_64 libssh2-1.4.2-3.el6_10.1.x86_64 nspr-4.21.0-1.el6_10.x86_64 nss-3.44.0-7.el6_10.x86_64 nss-util-3.44.0-1.el6_10.x86_64 openldap-2.4.40-16.el6.x86_64 openssl-1.0.1e-58.el6_10.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) Hangup detected on fd 0
error detected on stdin
A debugging session is active.

    Inferior 1 [process 32658] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]
Detaching from program: /usr/bin/fdfs_storaged, process 32658
[root@YNET ~]#

现在这个上有group1 group3 group4 正在占用CPU

@happyfish100
Copy link
Owner

你这台机器上只有一个 fdfs_storaged实例吧？
另外，为啥 gdb输出中断了呢？能否将输出重定向到一个临时文件中呢，比如：
echo 'thread apply all bt' | gdb attach -p $(pidof fdfs_storaged) > /tmp/fdfs_storaged.txt 2>&1

@zhangjijian111
Copy link
Author

我这台服务器总共起了4个fdfs_storaged group1 group2 group3 group4
执行命令之后貌似一样的
fdfs_storaged.txt

@happyfish100
Copy link
Owner

和之前的输出一样的。
你试试如下命令行：
gdb attach -p $(pidof fdfs_storaged) > /tmp/fdfs_storaged.txt 2>&1 << EndOfInput
thread apply all bt
quit
EndOfInput

@zhangjijian111
Copy link
Author

fdfs_storaged.txt
现在是group3 节点占用CPU

@happyfish100
Copy link
Owner

刚才看了输出，没有看到异常。
我们需要抓取fdfs_storaged的CPU异常时的调用堆栈，需要在fdfs_storaged的CPU异常那台服务器执行一下。

@zhangjijian111
Copy link
Author

我这个就是发发生异常的服务器抓取的日志 这个新的日志的是group3 group4这两个节点占用CPU的情况
fdfs_storaged.txt

@zhangjijian111
Copy link
Author

Thread 35 (Thread 0x7f4768fa3700 (LWP 31574)):
#0 0x00007f476848a2e3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f4768999e6c in ioevent_loop (pThreadData=0x17e1890, recv_notify_callback=, clean_up_callback=0x41dc38 <task_finish_clean_up>, continue_flag=0x658eea) at ioevent_loop.c:125
#2 0x00007f476875712e in worker_thread_entrance (arg=0x17e1020) at sf_service.c:318
#3 0x00007f4768bcad14 in start_thread (arg=0x7f4768fa3700) at pthread_create.c:308
#4 0x00007f4768489c4d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

我看好几个都在循环处理吧 不知道什么原因导致的

@happyfish100
Copy link
Owner

这个没问题的，堵塞在 epoll_wait 调用上。

@happyfish100
Copy link
Owner

出现异常时，可以用 top命令查看一下是哪个线程占用的CPU。
命令行如下：
top -H -p $(pidof fdfs_storaged)

@zhangjijian111
Copy link
Author

最近我发现我另一台服务器的Nginx中的mod_fastdfs.conf 配置的tracker地址有错误 改正后就没有发现占用CPU的问题了 不知道fastdfs同步文件的时候 是不是依赖mod_fastdfs.conf文件模块

@happyfish100
Copy link
Owner

mod_fastdfs.conf是FastDFS的nginx插件专用的，FDFS server端不会用到这个配置文件。

@zhangjijian111
Copy link
Author

image
是的 现在用的人多了又开始了

@happyfish100
Copy link
Owner

你看一下你们网络的重传率高吗？

@zhangjijian111
Copy link
Author

image
上面这个是发生CPU占用的网络信息
image
这个是另一台部署了fastdfs的网络信息

看着重传率貌似挺高的 这两台服务器是阿里云的ECS 貌似都开通了IPV6 不知道是不是这个导致的

@happyfish100
Copy link
Owner

FastDFS你们有跨机房部署或者公网访问吗？会不会是网络带宽的问题？

@happyfish100
Copy link
Owner

出现CPU异常的时候，用如下命令：
perf top

查看大约15秒后截屏发上来，谢谢！

@zhangjijian111
Copy link
Author

image

这是group1 group2两个节点发生异常的情况

@happyfish100
Copy link
Owner

正常情况下 perf top 截屏发一下，多谢！

@zhangjijian111
Copy link
Author

image

@happyfish100
Copy link
Owner

应该是客户端断开连接而Linux服务器端一直触发 epoll_wait调用的问题（一个请求几乎会占满一个CPU）。
这个问题是FastDFS多年遗留的一个bug，v6.9我在MacOS下测试没问题而没有在Linux下进行验证（在Linux下还是存在问题）。
这次应该是根治了。修复代码已提交，需要使用 libfastommon 和 libserverframe的master代码，重新编译和安装即可。

@zhangjijian111
Copy link
Author

我都重新下载libfastommon 和 libserverframe的master代码，重新编译后 重装了fastdfs 还是会占用CPU
这个同步数据是不是使用了pthread_cancel 来结束的 我查看了好多死锁吧
这是使用 gdb attach pid info thread 查看的
image

这是 thread apply all bt
image

@happyfish100
Copy link
Owner

问题还没解决，现象很诡异啊。
方便加我微信交流吗？你搜索 fastdfs 可以找到我

@JimmyPoor
Copy link

请问问题现在解决了吗？谢谢

@happyfish100
Copy link
Owner

据了解是在CentOS 6下使用某云提供的NAS存储会出现这个现象，而在CentOS 7及以上版本没有这个问题。

happyfish100 added a commit that referenced this issue Feb 12, 2023
@happyfish100
use epoll edge trigger to resolve github issues #608
1e209da
@happyfish100
Copy link
Owner

刚发布的 V6.9.4修复了这个问题，请大家尽快升级到最新的V6.9.4。

@cqu-linmu cqu-linmu mentioned this issue May 15, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@happyfish100 @JimmyPoor @zhangjijian111