Visor Processing Reports Empty Errors Detected field for message and msapproval tasks

[frrist]

Describe the bug:

The Errors Detected filed for message and msapproval tasks is "empty" ({}) when their status is ERROR:

912740,bafy2bzacebppacou5wgmp3simdxoth6yuackewmz3sseeon3sryin25v6gk3a,watch_1625688475,msapprovals,2021-07-07 21:00:00.008374,2021-07-07 21:00:00.008470,ERROR,,{}
912740,bafy2bzacebppacou5wgmp3simdxoth6yuackewmz3sseeon3sryin25v6gk3a,watch_1625688475,messages,2021-07-07 21:00:00.008374,2021-07-07 21:00:00.008469,ERROR,,{}

I believe the cause could be related to marshaling an array of errors to json when writing to the table.

Steps to Reproduce:

Run visor in daemon mode against mainnet

Visor Version: v0.7.1

[frrist]

related to #596

[frrist]

The new watch job we deployed with report name watch_1625688475 has encountered 1 error while processing tasks: messages and msapprovals -- these are both extracted by the message processor.
Their entries in the visor_processing_report table are empty as far as errors messages go see the {}
Query:

select *
from visor_processing_reports
where reporter = 'watch_1625688475' and status = 'ERROR'
order by height desc

Result:

912740,bafy2bzacebppacou5wgmp3simdxoth6yuackewmz3sseeon3sryin25v6gk3a,watch_1625688475,messages,2021-07-07 21:00:00.008374,2021-07-07 21:00:00.008469,ERROR,,{}
912740,bafy2bzacebppacou5wgmp3simdxoth6yuackewmz3sseeon3sryin25v6gk3a,watch_1625688475,msapprovals,2021-07-07 21:00:00.008374,2021-07-07 21:00:00.008470,ERROR,,{}
912738,bafy2bzacedbergvrromc7e4orymoxnx4wv7jrwbn2m36ak4uw4s6bfhhnz2yu,watch_1625688475,msapprovals,2021-07-07 20:59:00.512276,2021-07-07 20:59:00.512384,ERROR,,{}
912738,bafy2bzacedbergvrromc7e4orymoxnx4wv7jrwbn2m36ak4uw4s6bfhhnz2yu,watch_1625688475,messages,2021-07-07 20:59:00.512276,2021-07-07 20:59:00.512383,ERROR,,{}

I was able to find logs for these related to the failuer:

{"level":"error","ts":"2021-07-07T21:00:00.008Z","logger":"visor/chain","caller":"chain/indexer.go:214","msg":"failed to extract messages","height":912740,"error":"child tipset ({bafy2bzacea6w7hsmksq6yjwvgh4ude6fpatm3y4zvnr262l3qoy7fth7365yq,bafy2bzacealc77s7ldyctsscdrgaoeezt723ujbakluci3crpmkqrwkqmtyo4,bafy2bzacebcytkst5qq7yv2vnwyj4il42s5vggi6elstu2vmjcytw5ic2fsni,bafy2bzacebqtwg574z7gzmduczym53r66biz3ddt2t426wsf2xnaxi3itogds,bafy2bzaceam7t5xiiu2vbn6l5kwqtekedvmafhlg4zx7z7u3wxf5rcdwhc7sm,bafy2bzacecxsdxo7ucuensw2owgo6dagtbcaxo3oo6ypw57rudgqpv6m7fpo2}) is not on the same chain as parent ({bafy2bzacec6hmtvdgtpuhlfrrblwsyltyk7bmqrjq2firgnwkzepwx2pzsmyy,bafy2bzaceabtor3kworwfyeouimgenfqe4gup6q6lfoihjvbchhm4qtt53gda,bafy2bzacedv4u7xstz345squvwkag42sr7jfiprgtax4gmc6pc7l5lqrlnqd2,bafy2bzaced5n72p3k7xo77mjzbkewfdbtspfehnkiqbxiruerxnpgswaxtp3u,bafy2bzacebxqsutub5464jnbzlv7npzushn5wp7pxhgrmvhrryutdv5f6lbyg,bafy2bzacedpcb4z2qotusrc6eyi6lmhc77afuyd5w7gydvzkblwuzn3cx6n2u,bafy2bzaceb5du432p3vs53dojksc4wd25ei6hx65jzhnjscrwi6mlwb4cwbmw,bafy2bzacea6pfswjezghgcl4rm73m7i7kujrh7zao3va3hgvs2qlwrfwmwp5k,bafy2bzacecttvtlkegtzsrx6gwszl5alg47kmiamukqyg3qjvglphjy6fh3ws})","errorVerbose":"child tipset ({bafy2bzacea6w7hsmksq6yjwvgh4ude6fpatm3y4zvnr262l3qoy7fth7365yq,bafy2bzacealc77s7ldyctsscdrgaoeezt723ujbakluci3crpmkqrwkqmtyo4,bafy2bzacebcytkst5qq7yv2vnwyj4il42s5vggi6elstu2vmjcytw5ic2fsni,bafy2bzacebqtwg574z7gzmduczym53r66biz3ddt2t426wsf2xnaxi3itogds,bafy2bzaceam7t5xiiu2vbn6l5kwqtekedvmafhlg4zx7z7u3wxf5rcdwhc7sm,bafy2bzacecxsdxo7ucuensw2owgo6dagtbcaxo3oo6ypw57rudgqpv6m7fpo2}) is not on the same chain as parent ({bafy2bzacec6hmtvdgtpuhlfrrblwsyltyk7bmqrjq2firgnwkzepwx2pzsmyy,bafy2bzaceabtor3kworwfyeouimgenfqe4gup6q6lfoihjvbchhm4qtt53gda,bafy2bzacedv4u7xstz345squvwkag42sr7jfiprgtax4gmc6pc7l5lqrlnqd2,bafy2bzaced5n72p3k7xo77mjzbkewfdbtspfehnkiqbxiruerxnpgswaxtp3u,bafy2bzacebxqsutub5464jnbzlv7npzushn5wp7pxhgrmvhrryutdv5f6lbyg,bafy2bzacedpcb4z2qotusrc6eyi6lmhc77afuyd5w7gydvzkblwuzn3cx6n2u,bafy2bzaceb5du432p3vs53dojksc4wd25ei6hx65jzhnjscrwi6mlwb4cwbmw,bafy2bzacea6pfswjezghgcl4rm73m7i7kujrh7zao3va3hgvs2qlwrfwmwp5k,bafy2bzacecttvtlkegtzsrx6gwszl5alg47kmiamukqyg3qjvglphjy6fh3ws}):\n    github.com/filecoin-project/sentinel-visor/lens/util.GetExecutedAndBlockMessagesForTipset\n        /go/src/github.com/filecoin-project/sentinel-visor/lens/util/repo.go:251"}

{"level":"error","ts":"2021-07-07T20:59:00.512Z","logger":"visor/chain","caller":"chain/indexer.go:214","msg":"failed to extract messages","height":912738,"error":"child tipset ({bafy2bzacec6hmtvdgtpuhlfrrblwsyltyk7bmqrjq2firgnwkzepwx2pzsmyy,bafy2bzaceabtor3kworwfyeouimgenfqe4gup6q6lfoihjvbchhm4qtt53gda,bafy2bzacedv4u7xstz345squvwkag42sr7jfiprgtax4gmc6pc7l5lqrlnqd2,bafy2bzaced5n72p3k7xo77mjzbkewfdbtspfehnkiqbxiruerxnpgswaxtp3u,bafy2bzacebxqsutub5464jnbzlv7npzushn5wp7pxhgrmvhrryutdv5f6lbyg,bafy2bzacedpcb4z2qotusrc6eyi6lmhc77afuyd5w7gydvzkblwuzn3cx6n2u,bafy2bzaceb5du432p3vs53dojksc4wd25ei6hx65jzhnjscrwi6mlwb4cwbmw,bafy2bzacea6pfswjezghgcl4rm73m7i7kujrh7zao3va3hgvs2qlwrfwmwp5k,bafy2bzacecttvtlkegtzsrx6gwszl5alg47kmiamukqyg3qjvglphjy6fh3ws}) is not on the same chain as parent ({bafy2bzacednba2pob5drzb62mwdi7t22oewct7v2yr7fwzhwppb2auq4wwtdk,bafy2bzaceaij7klcd6zwiuv2zca2sxqs3gg2jvsyzsm6jlw6qby424nvvdqgq,bafy2bzaced3mr7myv3q4urhjydmevpzur3i6veocpu3azgi3pky7pzvefhwl2,bafy2bzacebjwfs5gwcaqbytwlb3kxctzcy7n3a4tddh6wrpo6g7k2bsgvckwm,bafy2bzaceb2lzrr4arzl45zlpsjvvwp4t6pxh45drjocqbli4msbixnhurqnu,bafy2bzaceaiofrmhrmpflaqx5cqrryn4od3aqqnudpwcepnz3nfde43twqbgq,bafy2bzaced42b3wtysqi7o43s4bhofvjdsdlkvxwqtyph5bquduqz5nw3ylqi})","errorVerbose":"child tipset ({bafy2bzacec6hmtvdgtpuhlfrrblwsyltyk7bmqrjq2firgnwkzepwx2pzsmyy,bafy2bzaceabtor3kworwfyeouimgenfqe4gup6q6lfoihjvbchhm4qtt53gda,bafy2bzacedv4u7xstz345squvwkag42sr7jfiprgtax4gmc6pc7l5lqrlnqd2,bafy2bzaced5n72p3k7xo77mjzbkewfdbtspfehnkiqbxiruerxnpgswaxtp3u,bafy2bzacebxqsutub5464jnbzlv7npzushn5wp7pxhgrmvhrryutdv5f6lbyg,bafy2bzacedpcb4z2qotusrc6eyi6lmhc77afuyd5w7gydvzkblwuzn3cx6n2u,bafy2bzaceb5du432p3vs53dojksc4wd25ei6hx65jzhnjscrwi6mlwb4cwbmw,bafy2bzacea6pfswjezghgcl4rm73m7i7kujrh7zao3va3hgvs2qlwrfwmwp5k,bafy2bzacecttvtlkegtzsrx6gwszl5alg47kmiamukqyg3qjvglphjy6fh3ws}) is not on the same chain as parent ({bafy2bzacednba2pob5drzb62mwdi7t22oewct7v2yr7fwzhwppb2auq4wwtdk,bafy2bzaceaij7klcd6zwiuv2zca2sxqs3gg2jvsyzsm6jlw6qby424nvvdqgq,bafy2bzaced3mr7myv3q4urhjydmevpzur3i6veocpu3azgi3pky7pzvefhwl2,bafy2bzacebjwfs5gwcaqbytwlb3kxctzcy7n3a4tddh6wrpo6g7k2bsgvckwm,bafy2bzaceb2lzrr4arzl45zlpsjvvwp4t6pxh45drjocqbli4msbixnhurqnu,bafy2bzaceaiofrmhrmpflaqx5cqrryn4od3aqqnudpwcepnz3nfde43twqbgq,bafy2bzaced42b3wtysqi7o43s4bhofvjdsdlkvxwqtyph5bquduqz5nw3ylqi}):\n    github.com/filecoin-project/sentinel-visor/lens/util.GetExecutedAndBlockMessagesForTipset\n        /go/src/github.com/filecoin-project/sentinel-visor/lens/util/repo.go:251"}

This seems to indicate that the Error field of the message processing report is failing to be persisted to the database as json and doing so silently.

[placer14]

FYI, results from gapfill and gapfind tasks also do not include error details when they are reported. Here's a CSV dump of a few of those records:

reporter	task	status	status_information	errors_detected
lily-production-mainnet-analysis-gapfill/gapfill-/t=Sin,f=1149120,t=1152240	actorstatesinit	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=Sma,f=1149120,t=1152240	actorstatesmarket	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	actorstatesmultisig	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	actorstatespower	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	actorstatesraw	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	actorstatesreward	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	actorstatesverifreg	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	implicitmessage	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	messages	ERROR		{}
lily-production-mainnet-analysis-gapfill/gapfill-/t=SmuSpoSraSreSveblchcoimmems,f=1149120,t=1152240	msapprovals	ERROR		{}