Board Report: Minutes not showing related reports other than Minutes

[shrayshray]

We started relating to Minutes all the reports which were acted on in that meeting and accounted for in the Minutes, and the Minutes approved in those Minutes (previous meeting) and the Minutes which approved these Minutes (next meeting).
For some reason, the reports related to the Minutes aren't showing - only the reports which type = Minutes show. Here is an example of this issue: https://boardagendas.metro.net/board-report/2019-0454/

Here's an example of where its working correctly (all related reports are showing): https://boardagendas.metro.net/board-report/2020-0714/

[fgomez828]

Hi @shrayshray! @hancush and I took a look at what was going on here, and it looks like we have some logic in the legistar scraper that results us only getting the matter relations with the highest value for the MatterRelationFlag property in legistar.

Here's the legistar matter info for the bill you linked to in this issue where the related_bills show up as expected: http://webapi.legistar.com/v1/metro/matters/7156/relations

And the matter info for the bill where the related_bills are not showing up as expected: http://webapi.legistar.com/v1/metro/matters/6008/relations

Is there a way to set all of the matter relations to have the same value for this flag?

[hancush]

@camachoo will look into how the matter relation flag is set. In the meantime, we will revise the Legistar scraper to grab all relations. We'll see how many duplicates (if any?) this causes.

[hancush]

Related issue: #227

PRs:

• Create a set of Matter ids opencivicdata/scrapers-us-municipal#169
• Filter relations based on MatterRelationFlag opencivicdata/python-legistar-scraper#47

[hancush]

I just deployed a change to the Legistar scraper that should capture all related board reports. Will check on this tomorrow (after the full scrape overnight) to confirm that all attachments have come through for https://boardagendas.metro.net/board-report/2019-0454/.

[hancush]

The scraper change wasn't captured in our scraper image due to Docker build caching. I resolved this in datamade/scrapers-us-municipal#60 and confirmed the image contains the latest version of the scraper. Keeping my eyeballs peeled for these attachments to come through overnight tonight.

[hancush]

@shrayshray Behold! Attachments have come through: https://boardagendas.metro.net/board-report/2019-0454/

Want to spot check a few other minutes files to ensure the fix repaired attachments more broadly?

[shrayshray]

@hancush I spot checked about 10 and they all look good - thank you!!

[hancush]

The correct relation is not always the max. Some chairs submit blank cover, then update previous version. Regina implemented a way to identify which version is the active version. I'll open a separate issue to check this, and close this.

[hancush]

Opened a separate issue about detecting the active version. For our purposes, we're only using the version to generate a distinct list of matter IDs from which to build relations. From that list, we capture matter file ID and intro date (in order to determine the legislative session it belongs to). @shrayshray, @camachoo – do either of these pieces of information vary between versions of a matter?

[shrayshray]

@hancush No. The matter file ID and intro date would be consistent across versions of a matter.

[hancush]

I will look up how we are handling getting the correct version at the board report level and update this thread. If it's capturing the correct version, then we can close this.

[hancush]

We use the MatterVersion attribute of an API Matter to determine which version of the text to retrieve.

Relevant links:

• Board Reports: Title not showing for some Reports #237
• Text with version arg opencivicdata/python-legistar-scraper#82

@shrayshray This approach seems to be working as expected. I think it's safe to close this!