Source does not recover from a "field type conflict" error

[w-le]

Source does not recover from a "field type conflict" error

In about 3-4 seperate deployments of Source, I've noticed that after a period of time it stops streaming data to Influx, but then works again after restarting Source.

I believe it may be due to the below type error, from which I think it never recovers. Prior to the error most log entries are publish messages, but following the error all log entries are healthchecks only.

level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.status_events message="redis pevent" pattern=status/mod-* channel=status/mod-HokAEHrr1rr/2410001.AnalogValue[476]
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 12:04:32 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"zone-G6Pq14teS4X","pos_area":"_"} fields={"pos_system":"sys-G6QRH-cB~tm","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_476_","2410001_AnalogValue_476_":0.44815385}
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 12:04:32 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"_","pos_area":"_"} fields={"pos_system":"sys-G7j2PfkboAX","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_476_","2410001_AnalogValue_476_":0.44815385}
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.status_events message="redis pevent" pattern=status/mod-* channel=status/mod-HokAEHrr1rr/2410001.AnalogValue[482]
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 12:04:32 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"zone-G6Pq14teS4X","pos_area":"_"} fields={"pos_system":"sys-G6QRH-cB~tm","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_482_","2410001_AnalogValue_482_":5.3648586}
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 12:04:32 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"_","pos_area":"_"} fields={"pos_system":"sys-G7j2PfkboAX","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_482_","2410001_AnalogValue_482_":5.3648586}
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.status_events message="redis pevent" pattern=status/mod-* channel=status/mod-HokAEHrr1rr/2410001.AnalogValue[475]
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 12:04:32 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"zone-G6Pq14teS4X","pos_area":"_"} fields={"pos_system":"sys-G6QRH-cB~tm","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_475_","2410001_AnalogValue_475_":33351.28}
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 12:04:32 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"_","pos_area":"_"} fields={"pos_system":"sys-G7j2PfkboAX","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_475_","2410001_AnalogValue_475_":33351.28}
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.status_events message="redis pevent" pattern=status/mod-* channel=status/mod-HokAEHrr1rr/2410001.AnalogValue[474]
level=[D] time=2021-08-10T12:04:32Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 12:04:32 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"zone-G6Pq14teS4X","pos_area":"_"} fields={"pos_system":"sys-G6QRH-cB~tm","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_474_","2410001_AnalogValue_474_":408214}
Unhandled exception in spawn: failure writing points to database: partial write: field type conflict: input field "2410001_AnalogValue_474_" on measurement "BACnet" is type integer, already exists as type float dropped=1 (Flux::ClientError)
  from usr/share/crystal/src/string.cr:2633:31 in 'publish'
  from app/src/source/publishing/publisher.cr:30:11 in '->'
  from usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???
level=[I] time=2021-08-10T12:04:38Z program=source source=action-controller client_ip=10.240.0.5 request_id=bf9b0c3e-afbe-41df-991c-5332c93339e3 event=response method=GET path=/api/source/v1/ status=200 duration=719.63µs
level=[I] time=2021-08-10T12:04:41Z program=source source=action-controller client_ip=10.240.0.5 request_id=dec63deb-7c69-4565-9ba5-7eb713255955 event=response method=GET path=/api/source/v1/ status=200 duration=732.13µs
level=[I] time=2021-08-10T12:04:48Z program=source source=action-controller client_ip=10.240.0.5 request_id=292c863c-a5fc-48aa-bfc9-76492f0c76c3 event=response method=GET path=/api/source/v1/ status=200 duration=834.94µs
level=[I] time=2021-08-10T12:04:51Z program=source source=action-controller client_ip=10.240.0.5 request_id=c51676ad-5f5e-4e48-ab59-8d900055d23f event=response method=GET path=/api/source/v1/ status=200 duration=753.54µs
level=[I] time=2021-08-10T12:04:58Z program=source source=action-controller client_ip=10.240.0.5 request_id=881171fd-9f77-46cf-9de1-587db8f3f7d1 event=response method=GET path=/api/source/v1/ status=200 duration=913.54µs
level=[I] time=2021-08-10T12:05:01Z program=source source=action-controller client_ip=10.240.0.5 request_id=020b4b9d-217d-4852-9993-bdd9efe16d7d event=response method=GET path=/api/source/v1/ status=200 duration=776.74µs
level=[I] time=2021-08-10T12:05:08Z program=source source=action-controller client_ip=10.240.0.5 request_id=914677fd-37b5-4d38-8277-bcdf344cf721 event=response method=GET path=/api/source/v1/ status=200 duration=1.02ms
level=[I] time=2021-08-10T12:05:11Z program=source source=action-controller client_ip=10.240.0.5 request_id=f90d50a5-7ac4-4498-be32-e11897b503e8 event=response method=GET path=/api/source/v1/ status=200 duration=1.14ms
level=[I] time=2021-08-10T12:05:18Z program=source source=action-controller client_ip=10.240.0.5 request_id=54472a07-dc56-44d7-8267-614323d43b1b event=response method=GET path=/api/source/v1/ status=200 duration=833.94µs
level=[I] time=2021-08-10T12:05:21Z program=source source=action-controller client_ip=10.240.0.5 request_id=e3a8a63b-0687-4f18-9208-b492a3c60cc7 event=response method=GET path=/api/source/v1/ status=200 duration=1.02ms

Here is a similar occurrence (with the same consequence) on the same instance, but related to a different Driver type:

level=[D] time=2021-08-10T11:45:24Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 11:45:24 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"zone-G6Pq14teS4X","pos_area":"_"} fields={"pos_system":"sys-G6QRH-cB~tm","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_961_","2410001_AnalogValue_961_":0.76573217}
level=[D] time=2021-08-10T11:45:24Z program=source source=place_os.source.publisher measurement=BACnet timestamp=2021-08-10 11:45:24 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"_","pos_area":"_"} fields={"pos_system":"sys-G7j2PfkboAX","pos_driver":"driver-Hok6~MJJIth","pos_index":1,"pos_key":"2410001_AnalogValue_961_","2410001_AnalogValue_961_":0.76573217}
level=[D] time=2021-08-10T11:45:25Z program=source source=place_os.source.status_events message="redis pevent" pattern=status/mod-* channel=status/mod-G7j1d_yQqbx/zone-G6Q2rNI~zPF:sensors
level=[D] time=2021-08-10T11:45:25Z program=source source=place_os.source.publisher measurement=AreaManagement timestamp=2021-08-10 11:45:25 UTC tags={"pos_org":"zone-G6PpWmOci3j","pos_building":"zone-G6Pxx7b3e6K","pos_level":"zone-G6Q2rNI~zPF","pos_area":"_","pos_uniq":"0"} fields={"pos_system":"sys-G7j2PfkboAX","pos_driver":"driver-G7Y8RBP6kzp","pos_index":1,"pos_key":"zone_G6Q2rNI_zPF_sensors","status":"normal","type":"humidity","value":69.30000305175781,"last_seen":1628517768,"mac":"2410001","id":"AnalogValue[236]","name":"device2410001: supplyairhumidity_5","location":"sensor","xloc":0.1505,"yloc":0.5024,"building":"zone-G6Pxx7b3e6K","level":"zone-G6Q2rNI~zPF"}
level=[D] time=2021-08-10T11:45:25Z program=source source=place_os.source.status_events message="redis pevent" pattern=status/mod-* channel=status/mod-G7j1d_yQqbx/zone-G6Pq14teS4X:sensors
Unhandled exception in spawn: failure writing points to database: partial write: field type conflict: input field "level" on measurement "AreaManagement" is type string, already exists as type float dropped=1 (Flux::ClientError)
  from usr/share/crystal/src/string.cr:2633:31 in 'publish'
  from app/src/source/publishing/publisher.cr:30:11 in '->'
  from usr/share/crystal/src/primitives.cr:255:3 in 'run'
  from ???