Warewulfd panic after adding 2nd node to the cluster configuration #468

BeHom · 2022-06-24T13:53:19Z

Used latest version 4.3.0-1.git_235c23c

Problem descriptions.
while adding a 2nd node to the cluster using wwctl command the warewulfd panic.
Entries in the nodes.conf are made but warewulfd failed silent.
Restart failed as well, without any error message on the console.
Only a status request will show the problem (need to have an error message for the user).
Restarts are not possible unless the new added node in the nodes.conf are deleted
After deleting the new added node in the nodes.conf warewuld will start normal and the 1st node in the cluster can boot.

start of console log #################
[root@emil log]0 # wwctl node add hpc3 --ipaddr 10.25.121.13 --discoverable
Added node: hpc3
[root@emil log]0 # systemctl status warewulfd
● warewulfd.service - Warewulf cluster management daemon
Loaded: loaded (/usr/lib/systemd/system/warewulfd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-06-24 15:32:20 CEST; 15s ago
Docs: https://warewulf.hpcng.org/
Process: 1142324 ExecStart=/usr/bin/wwctl server start (code=exited, status=2)
Main PID: 1142324 (code=exited, status=2)

Jun 24 15:32:20 emil.k-net.ka.de systemd[1]: warewulfd.service: Service RestartSec=100ms expired, scheduling restart.
Jun 24 15:32:20 emil.k-net.ka.de systemd[1]: warewulfd.service: Scheduled restart job, restart counter is at 6.
Jun 24 15:32:20 emil.k-net.ka.de systemd[1]: Stopped Warewulf cluster management daemon.
Jun 24 15:32:20 emil.k-net.ka.de systemd[1]: warewulfd.service: Start request repeated too quickly.
Jun 24 15:32:20 emil.k-net.ka.de systemd[1]: warewulfd.service: Failed with result 'exit-code'.
Jun 24 15:32:20 emil.k-net.ka.de systemd[1]: Failed to start Warewulf cluster management daemon.
[root@emil log]3 # systemctl start warewulfd
[root@emil log]0 # systemctl status warewulfd
● warewulfd.service - Warewulf cluster management daemon
Loaded: loaded (/usr/lib/systemd/system/warewulfd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2022-06-24 15:32:47 CEST; 10s ago
Docs: https://warewulf.hpcng.org/
Process: 1142586 ExecStart=/usr/bin/wwctl server start (code=exited, status=2)
Main PID: 1142586 (code=exited, status=2)

Jun 24 15:32:47 emil.k-net.ka.de systemd[1]: warewulfd.service: Service RestartSec=100ms expired, scheduling restart.
Jun 24 15:32:47 emil.k-net.ka.de systemd[1]: warewulfd.service: Scheduled restart job, restart counter is at 5.
Jun 24 15:32:47 emil.k-net.ka.de systemd[1]: Stopped Warewulf cluster management daemon.
Jun 24 15:32:47 emil.k-net.ka.de systemd[1]: warewulfd.service: Start request repeated too quickly.
Jun 24 15:32:47 emil.k-net.ka.de systemd[1]: warewulfd.service: Failed with result 'exit-code'.
Jun 24 15:32:47 emil.k-net.ka.de systemd[1]: Failed to start Warewulf cluster management daemon.

end of console log############

it seems that the entry in the nodes.conf is ok but warewulf failed to use them
"nodes.conf Extract "
hpc3:
discoverable: "true"
profiles:
- default
network devices:
default:
onboot: "true"
ipaddr: 10.25.121.13

[root@emil log]0 # cat warewulfd.log
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/hpcng/warewulf/internal/pkg/node.(*nodeYaml).FindAllNodes(0xc00069f338)
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/node/constructors.go:262 +0x43f1
github.com/hpcng/warewulf/internal/pkg/warewulfd.loadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:36 +0x56
github.com/hpcng/warewulf/internal/pkg/warewulfd.LoadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:25 +0x6d
github.com/hpcng/warewulf/internal/pkg/warewulfd.RunServer()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/warewulfd.go:42 +0xd9
github.com/hpcng/warewulf/internal/pkg/warewulfd.DaemonStart()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/daemon.go:74 +0x65
github.com/hpcng/warewulf/internal/app/wwctl/server/start.CobraRunE(0x15ecb40, {0xd420cb, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/internal/app/wwctl/server/start/main.go:20 +0xfd
github.com/spf13/cobra.(*Command).execute(0x15ecb40, {0x1691d20, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:850 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x15e3d80)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:895
main.main()
/builddir/build/BUILD/warewulf-4.3.0/cmd/wwctl/main.go:13 +0x25
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/hpcng/warewulf/internal/pkg/node.(*nodeYaml).FindAllNodes(0xc00061f338)
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/node/constructors.go:262 +0x43f1
github.com/hpcng/warewulf/internal/pkg/warewulfd.loadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:36 +0x56
github.com/hpcng/warewulf/internal/pkg/warewulfd.LoadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:25 +0x6d
github.com/hpcng/warewulf/internal/pkg/warewulfd.RunServer()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/warewulfd.go:42 +0xd9
github.com/hpcng/warewulf/internal/pkg/warewulfd.DaemonStart()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/daemon.go:74 +0x65
github.com/hpcng/warewulf/internal/app/wwctl/server/start.CobraRunE(0x15ecb40, {0xd420cb, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/internal/app/wwctl/server/start/main.go:20 +0xfd
github.com/spf13/cobra.(*Command).execute(0x15ecb40, {0x1691d20, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:850 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x15e3d80)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:895
main.main()
/builddir/build/BUILD/warewulf-4.3.0/cmd/wwctl/main.go:13 +0x25
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/hpcng/warewulf/internal/pkg/node.(*nodeYaml).FindAllNodes(0xc00069f338)
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/node/constructors.go:262 +0x43f1
github.com/hpcng/warewulf/internal/pkg/warewulfd.loadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:36 +0x56
github.com/hpcng/warewulf/internal/pkg/warewulfd.LoadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:25 +0x6d
github.com/hpcng/warewulf/internal/pkg/warewulfd.RunServer()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/warewulfd.go:42 +0xd9
github.com/hpcng/warewulf/internal/pkg/warewulfd.DaemonStart()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/daemon.go:74 +0x65
github.com/hpcng/warewulf/internal/app/wwctl/server/start.CobraRunE(0x15ecb40, {0xd420cb, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/internal/app/wwctl/server/start/main.go:20 +0xfd
github.com/spf13/cobra.(*Command).execute(0x15ecb40, {0x1691d20, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:850 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x15e3d80)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:895
main.main()
/builddir/build/BUILD/warewulf-4.3.0/cmd/wwctl/main.go:13 +0x25
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/hpcng/warewulf/internal/pkg/node.(*nodeYaml).FindAllNodes(0xc00025f338)
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/node/constructors.go:262 +0x43f1
github.com/hpcng/warewulf/internal/pkg/warewulfd.loadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:36 +0x56
github.com/hpcng/warewulf/internal/pkg/warewulfd.LoadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:25 +0x6d
github.com/hpcng/warewulf/internal/pkg/warewulfd.RunServer()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/warewulfd.go:42 +0xd9
github.com/hpcng/warewulf/internal/pkg/warewulfd.DaemonStart()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/daemon.go:74 +0x65
github.com/hpcng/warewulf/internal/app/wwctl/server/start.CobraRunE(0x15ecb40, {0xd420cb, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/internal/app/wwctl/server/start/main.go:20 +0xfd
github.com/spf13/cobra.(*Command).execute(0x15ecb40, {0x1691d20, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:850 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x15e3d80)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:895
main.main()
/builddir/build/BUILD/warewulf-4.3.0/cmd/wwctl/main.go:13 +0x25
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/hpcng/warewulf/internal/pkg/node.(*nodeYaml).FindAllNodes(0xc00061f338)
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/node/constructors.go:262 +0x43f1
github.com/hpcng/warewulf/internal/pkg/warewulfd.loadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:36 +0x56
github.com/hpcng/warewulf/internal/pkg/warewulfd.LoadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:25 +0x6d
github.com/hpcng/warewulf/internal/pkg/warewulfd.RunServer()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/warewulfd.go:42 +0xd9
github.com/hpcng/warewulf/internal/pkg/warewulfd.DaemonStart()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/daemon.go:74 +0x65
github.com/hpcng/warewulf/internal/app/wwctl/server/start.CobraRunE(0x15ecb40, {0xd420cb, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/internal/app/wwctl/server/start/main.go:20 +0xfd
github.com/spf13/cobra.(*Command).execute(0x15ecb40, {0x1691d20, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:850 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x15e3d80)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:895
main.main()
/builddir/build/BUILD/warewulf-4.3.0/cmd/wwctl/main.go:13 +0x25
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/hpcng/warewulf/internal/pkg/node.(*nodeYaml).FindAllNodes(0xc00069f338)
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/node/constructors.go:262 +0x43f1
github.com/hpcng/warewulf/internal/pkg/warewulfd.loadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:36 +0x56
github.com/hpcng/warewulf/internal/pkg/warewulfd.LoadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:25 +0x6d
github.com/hpcng/warewulf/internal/pkg/warewulfd.RunServer()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/warewulfd.go:42 +0xd9
github.com/hpcng/warewulf/internal/pkg/warewulfd.DaemonStart()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/daemon.go:74 +0x65
github.com/hpcng/warewulf/internal/app/wwctl/server/start.CobraRunE(0x15ecb40, {0xd420cb, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/internal/app/wwctl/server/start/main.go:20 +0xfd
github.com/spf13/cobra.(*Command).execute(0x15ecb40, {0x1691d20, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:850 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x15e3d80)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:895
main.main()
/builddir/build/BUILD/warewulf-4.3.0/cmd/wwctl/main.go:13 +0x25
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/hpcng/warewulf/internal/pkg/node.(*nodeYaml).FindAllNodes(0xc00061f338)
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/node/constructors.go:262 +0x43f1
github.com/hpcng/warewulf/internal/pkg/warewulfd.loadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:36 +0x56
github.com/hpcng/warewulf/internal/pkg/warewulfd.LoadNodeDB()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/nodedb.go:25 +0x6d
github.com/hpcng/warewulf/internal/pkg/warewulfd.RunServer()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/warewulfd.go:42 +0xd9
github.com/hpcng/warewulf/internal/pkg/warewulfd.DaemonStart()
/builddir/build/BUILD/warewulf-4.3.0/internal/pkg/warewulfd/daemon.go:74 +0x65
github.com/hpcng/warewulf/internal/app/wwctl/server/start.CobraRunE(0x15ecb40, {0xd420cb, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/internal/app/wwctl/server/start/main.go:20 +0xfd
github.com/spf13/cobra.(*Command).execute(0x15ecb40, {0x1691d20, 0x0, 0x0})
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:850 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0x15e3d80)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:958 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/builddir/build/BUILD/warewulf-4.3.0/vendor/github.com/spf13/cobra/command.go:895
main.main()
/builddir/build/BUILD/warewulf-4.3.0/cmd/wwctl/main.go:13 +0x25

mslacken · 2022-06-24T14:49:35Z

I can confirm this bug. It is triggered as soon as Tags are defined in a profile.

mslacken · 2022-06-24T15:01:56Z

@BeHom This bug is only triggered if the node is added without the network which does have a tag defined. So following command should work

wwctl node add hpc03 --netname default -I 10.25.121.13

On node add you define the network and its associated ip address -I switch so that nodes.conf has the correct yaml entry.
I hope this workaround helps.

BeHom · 2022-06-24T15:12:42Z

many thanks for the quick support!
wwctl node add hpc03 --netname default -I 10.25.121.13 did not work.
warewulfd will silent fail.
Deleting the tags seems to fix the problem. Will further test it and perhaps using specific ifcg-.conf files for interfaces needing the MTU tag.

BeHom · 2022-06-24T15:20:02Z

2nd system is booting.

mslacken · 2022-06-24T15:27:21Z

Hope I fixed it, see the two PRs above.
I will call it a day now.

BeHom changed the title ~~Warwulfd panic after adding 2nd node to the cluster configuration~~ Warewulfd panic after adding 2nd node to the cluster configuration Jun 24, 2022

mslacken added bug Something isn't working backport:4.3.x backport to 4.3.x labels Jun 24, 2022

This was referenced Jun 24, 2022

fix nil map for networks #469

Merged

fix nil map for networks #470

Merged

mslacken closed this as completed in #469 Jul 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Warewulfd panic after adding 2nd node to the cluster configuration #468

Warewulfd panic after adding 2nd node to the cluster configuration #468

BeHom commented Jun 24, 2022

mslacken commented Jun 24, 2022

mslacken commented Jun 24, 2022

BeHom commented Jun 24, 2022

BeHom commented Jun 24, 2022

mslacken commented Jun 24, 2022

Warewulfd panic after adding 2nd node to the cluster configuration #468

Warewulfd panic after adding 2nd node to the cluster configuration #468

Comments

BeHom commented Jun 24, 2022

mslacken commented Jun 24, 2022

mslacken commented Jun 24, 2022

BeHom commented Jun 24, 2022

BeHom commented Jun 24, 2022

mslacken commented Jun 24, 2022