Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic from cluster.check #2905

Closed
Jackbennett opened this issue Apr 12, 2022 · 1 comment
Closed

Panic from cluster.check #2905

Jackbennett opened this issue Apr 12, 2022 · 1 comment

Comments

@Jackbennett
Copy link

Jackbennett commented Apr 12, 2022

Describe the bug
cluster.check from weed shell errors with a stack trace.

System Setup

  • command weed -v 4 -config_dir ./config/ shell
  • OS version VERSION="18.04.3 LTS (Bionic Beaver)"
  • output of weed version version 30GB 2.98 c6ec5269f4b34d79ab8e13050623501b8befda32 linux amd64
  • scaffold of filer.toml in use.
[leveldb2]
# local on disk, mostly for simple single-machine setup, fairly scalable
# faster than previous leveldb, recommended.
enabled = true
dir = "./filerldb2"                    # directory to store level db files
> /srv/seaweedfs/production$ weed -v 4 -config_dir ./config/ shell
I0412 10:21:03 21181 config.go:59] Reading security.toml from /srv/seaweedfs/production/config/security.toml
I0412 10:21:03 21181 config.go:46] Reading : Config File "shell" Not Found in "[/srv/seaweedfs/production/config /srv/seaweedfs/production /home/jackben/.seaweedfs /usr/local/etc/seaweedfs /etc/seaweedfs]"
I0412 10:21:03 21181 masterclient.go:56] adminShell masterClient bootstraps with masters map[localhost:9333:localhost:9333]
I0412 10:21:03 21181 masterclient.go:104] adminShell masterClient Connecting to master localhost:9333
I0412 10:21:03 21181 masterclient.go:127] adminShell masterClient Connected to localhost:9333
master: localhost:9333 filers: [10.9.8.21:8888]
> cluster.check
Topology volumeSizeLimit:2048 MB hdd(volume:0/399 active:0 free:399 remote:0)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1d88008]

goroutine 1 [running]:
github.com/chrislusf/seaweedfs/weed/shell.(*commandClusterCheck).Do(0xc000175cc0?, {0x42e97c8, 0x0, 0x0}, 0xc0007343c0, {0x2e062a0, 0xc00013a008})
	/github/workspace/weed/shell/command_cluster_check.go:51 +0x248
github.com/chrislusf/seaweedfs/weed/shell.processEachCmd(0xc00053ffe0?, {0xc00053ffe0, 0xd}, 0x1?)
	/github/workspace/weed/shell/shell_liner.go:136 +0x3a7
github.com/chrislusf/seaweedfs/weed/shell.RunShell({0xc000532490, {0x2e04600, 0xc00000e1f8}, {0x0, 0x0}, 0x0, {0x0, 0x0}, {0x27da260, 0x1}})
	/github/workspace/weed/shell/shell_liner.go:105 +0x525
github.com/chrislusf/seaweedfs/weed/command.runShell(0x4295358?, {0xc0001181d0?, 0x0?, 0x0?})
	/github/workspace/weed/command/shell.go:59 +0x2d8
main.main()
	/github/workspace/weed/weed.go:81 +0x383

Additional context
Really trying to filer.sync from an initial weed server test setup, but err from weed -config_dir ./config/ filer.sync -isActivePassive -a localhost:9999 -b localhost:8888 results in E0412 10:37:07 25622 filer_sync.go:105] sync from localhost:9999 to localhost:8888: GetFilerConfiguration localhost:9999: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake" so I'm checking the cluster is OK before getting into that error.

Also, this the security.toml note redactedN are all different hashes. If they/some need to be the same that's not in documentation I found.

# Put this file to one of the location, with descending priority
#    ./security.toml
#    $HOME/.seaweedfs/security.toml
#    /etc/seaweedfs/security.toml
# this file is read by master, volume server, and filer

# this jwt signing key is read by master and volume server, and it is used for write operations:
# - the Master server generates the JWT, which can be used to write a certain file on a volume server
# - the Volume server validates the JWT on writing
# the jwt defaults to expire after 10 seconds.
[jwt.signing]
key = "redacted1"
expires_after_seconds = 10           # seconds

# by default, if the signing key above is set, the Volume UI over HTTP is disabled.
# by setting ui.access to true, you can re-enable the Volume UI. Despite
# some information leakage (as the UI is not authenticated), this should not
# pose a security risk.
[access]
ui = true

# this jwt signing key is read by master and volume server, and it is used for read operations:
# - the Master server generates the JWT, which can be used to read a certain file on a volume server
# - the Volume server validates the JWT on reading
# NOTE: jwt for read is only supported with master+volume setup. Filer does not support this mode.
[jwt.signing.read]
key = "redacted2"
expires_after_seconds = 10           # seconds


# If this JWT key is configured, Filer only accepts writes over HTTP if they are signed with this JWT:
# - f.e. the S3 API Shim generates the JWT
# - the Filer server validates the JWT on writing
# the jwt defaults to expire after 10 seconds.
[jwt.filer_signing]
key = "redacted3"
expires_after_seconds = 10           # seconds

# If this JWT key is configured, Filer only accepts reads over HTTP if they are signed with this JWT:
# - f.e. the S3 API Shim generates the JWT
# - the Filer server validates the JWT on writing
# the jwt defaults to expire after 10 seconds.
[jwt.filer_signing.read]
key = "redacted4"
expires_after_seconds = 10           # seconds

# all grpc tls authentications are mutual
# the values for the following ca, cert, and key are paths to the PERM files.
# the host name is not checked, so the PERM files can be shared.
[grpc]
ca = "/srv/seaweedfs/production/CertificateAuthority/out/SeaweedFS_CA.crt"
# Set wildcard domain for enable TLS authentication by common names
allowed_wildcard_domain = "" # .mycompany.com

[grpc.volume]
cert = "/srv/seaweedfs/production/CertificateAuthority/out/volume01.crt"
key = "/srv/seaweedfs/production/CertificateAuthority/out/volume01.key"
allowed_commonNames = ""    # comma-separated SSL certificate common names

[grpc.master]
cert = "/srv/seaweedfs/production/CertificateAuthority/out/master01.crt"
key = "/srv/seaweedfs/production/CertificateAuthority/out/master01.key"
allowed_commonNames = ""    # comma-separated SSL certificate common names

[grpc.filer]
cert = "/srv/seaweedfs/production/CertificateAuthority/out/filer01.crt"
key = "/srv/seaweedfs/production/CertificateAuthority/out/filer01.key"
allowed_commonNames = ""    # comma-separated SSL certificate common names

[grpc.msg_broker]
cert = ""
key = ""
allowed_commonNames = ""    # comma-separated SSL certificate common names

# use this for any place needs a grpc client
# i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload"
[grpc.client]
cert = "/srv/seaweedfs/production/CertificateAuthority/out/client01.crt"
key = "/srv/seaweedfs/production/CertificateAuthority/out/client01.key"

# volume server https options
# Note: work in progress!
#     this does not work with other clients, e.g., "weed filer|mount" etc, yet.
[https.client]
enabled = true

[https.volume]
cert = ""
key = ""
ca = ""

[https.master]
cert = ""
key = ""
ca = ""
@chrislusf
Copy link
Collaborator

Thanks for the detailed report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants