Don't clear args, rely on delete deregister_callbacks

[SimonDanisch]

We did clear the arguments of a plot incorrectly.
I think this is a leftover from the refactor, where the ownership of the observables in plot.arg changed (before it was always created just for the plot).
Needs some tests, to make sure we don't introduce any memory leaks with this change.
Closes #3542

[MakieBot]

Compile Times benchmark

Note, that these numbers may fluctuate on the CI servers, so take them with a grain of salt. All benchmark results are based on the mean time and negative percent mean faster than the base branch. Note, that GLMakie + WGLMakie run on an emulated GPU, so the runtime benchmark is much slower. Results are from running:

using_time = @ctime using Backend
# Compile time
create_time = @ctime fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @ctime Makie.colorbuffer(display(fig))
# Runtime
create_time = @benchmark fig = scatter(1:4; color=1:4, colormap=:turbo, markersize=20, visible=true)
display_time = @benchmark Makie.colorbuffer(fig)

	using	create	display	create	display
GLMakie	3.41s (3.40, 3.42) 0.01+-	388.32ms (381.98, 396.25) 6.25+-	467.57ms (461.50, 473.59) 4.34+-	7.16ms (7.05, 7.27) 0.07+-	25.35ms (25.30, 25.41) 0.03+-
master	3.40s (3.38, 3.42) 0.01+-	396.28ms (391.58, 399.93) 2.85+-	476.17ms (468.02, 515.36) 17.35+-	7.22ms (7.14, 7.30) 0.06+-	25.37ms (25.33, 25.48) 0.05+-
evaluation	1.00x invariant, 0.01s (0.97d, 0.10p, 0.01std)	1.02x faster ✓, -7.96ms (-1.64d, 0.01p, 4.55std)	1.02x invariant, -8.6ms (-0.68d, 0.25p, 10.84std)	1.01x invariant, -0.06ms (-1.00d, 0.09p, 0.06std)	1.00x invariant, -0.03ms (-0.65d, 0.25p, 0.04std)
CairoMakie	3.09s (3.03, 3.20) 0.06+-	330.41ms (321.02, 345.86) 8.21+-	143.38ms (139.18, 147.07) 2.77+-	7.30ms (7.14, 7.45) 0.11+-	607.38μs (605.77, 609.10) 1.25+-
master	3.08s (3.02, 3.16) 0.06+-	330.01ms (322.77, 339.73) 6.54+-	144.26ms (140.45, 149.27) 3.57+-	7.29ms (7.15, 7.49) 0.14+-	609.04μs (604.22, 616.76) 3.86+-
evaluation	1.00x invariant, 0.01s (0.20d, 0.71p, 0.06std)	1.00x invariant, 0.4ms (0.05d, 0.92p, 7.37std)	1.01x invariant, -0.88ms (-0.28d, 0.62p, 3.17std)	1.00x invariant, 0.01ms (0.09d, 0.86p, 0.13std)	1.00x invariant, -1.66μs (-0.58d, 0.31p, 2.55std)
WGLMakie	3.72s (3.63, 3.82) 0.07+-	324.84ms (315.25, 335.19) 7.84+-	8.91s (8.70, 9.17) 0.15+-	9.15ms (8.99, 9.38) 0.13+-	73.85ms (68.60, 79.86) 4.78+-
master	3.70s (3.62, 3.78) 0.05+-	321.84ms (315.73, 331.13) 5.57+-	8.86s (8.74, 8.96) 0.09+-	9.64ms (8.96, 12.32) 1.20+-	70.94ms (67.65, 79.40) 4.27+-
evaluation	0.99x invariant, 0.02s (0.33d, 0.55p, 0.06std)	0.99x invariant, 2.99ms (0.44d, 0.43p, 6.70std)	0.99x invariant, 0.05s (0.40d, 0.48p, 0.12std)	1.05x noisy🤷‍♀️, -0.49ms (-0.58d, 0.32p, 0.66std)	0.96x invariant, 2.91ms (0.64d, 0.25p, 4.52std)