Fix vectorfield increase after each call

[hoxbro]

Fixes #5399

[douglasmacdonald]

Hope this message is not an abuse of the process, but I just want to say, "thank you", that was so quick.

[codecov-commenter]

Codecov Report

Merging #5400 (0116ce5) into master (8b2ee78) will decrease coverage by 0.00%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master    #5400      +/-   ##
==========================================
- Coverage   88.07%   88.07%   -0.01%     
==========================================
  Files         301      301              
  Lines       61969    61991      +22     
==========================================
+ Hits        54579    54596      +17     
- Misses       7390     7395       +5     

Impacted Files	Coverage Δ
holoviews/plotting/bokeh/chart.py	88.61% <50.00%> (ø)
holoviews/util/transform.py	85.25% <0.00%> (-0.75%)	⬇️
holoviews/core/data/pandas.py	93.30% <0.00%> (-0.10%)	⬇️
holoviews/tests/core/test_utils.py	99.20% <0.00%> (+<0.01%)	⬆️
holoviews/core/util.py	86.06% <0.00%> (+0.03%)	⬆️
holoviews/plotting/mpl/element.py	90.08% <0.00%> (+0.05%)	⬆️
...iews/tests/plotting/matplotlib/test_elementplot.py	98.42% <0.00%> (+0.07%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[jlstevens]

Looks good to me! Merging.