Table github_workflow fails with error when we pass all repositories

[madhushreeray30]

Describe the bug

> select * from github_workflow where repository_full_name in (select name_with_owner from github_my_repository)

Error: failed to populate column 'pipeline': yaml: unmarshal errors:
  line 87: cannot unmarshal !!str `${{ fro...` into []interface {} (SQLSTATE HV000)

+-----------------------------------------------+---------------------------------------------------+----------+------------------------------------------------------------+------------------------------------>
| repository_full_name                          | name                                              | id       | path                                                       | badge_url                          >
+-----------------------------------------------+---------------------------------------------------+----------+------------------------------------------------------------+------------------------------------>
| turbot/steampipe-mod-azure-insights           | Sync Labels                                       | 00000000 | .github/workflows/sync-labels.yml                          | https://github.com/turbot/steampipe>
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
| turbot/steampipe-mod-azure-insights           | Stale Issues and PRs                              | 00000000 | .github/workflows/stale.yml                                | https://github.com/turbot/steampipe>
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >
|                                               |                                                   |          |                                                            |                                    >

[gabrielsoltz]

Same here. It always fails when I pass all repositories.

[ParthaI]

Hello @madhushreeray30 @gabrielsoltz, Thanks for reporting the issue to us.

I was giving a dry-run to the table and was able to reproduce the error.
At first glance, I can see the error is due to a job/workflow having an iteration over a dynamic value.

E:g.
We have a workflow that implements a matrix strategy (matrix: ${{ fromJson(needs.get-wip-branches-set-matrix-input.outputs.wipBranches) }}) the value is dynamic. In such a case, the table throws an error because the table code can't parse the value ${{ fromJson(needs.get-wip-branches-set-matrix-input.outputs.wipBranches) }}.

I will have a look deeper into it and let you know.

Thanks!

[gabrielsoltz]

Do you have any progress on this one @ParthaI ?

[misraved]

@gabrielsoltz @ParthaI

I have been investigating the recent concerns and it appears that the root of the problem may lie with the ArgonSecurity Pipeline Parser package. Specifically, I am referring to the issue detailed here. In an attempt to address this, I updated the package to the latest version (1.3.6), but unfortunately, it seems that this update has introduced some breaking changes.

[ParthaI]

Hello @gabrielsoltz,

My apologies for the delay in resolving this issue.

I've implemented a fix addressing the problem and submitted it in this PR.

The issue stemmed from workflows or jobs iterating over dynamic values, as highlighted in the comment above.

After conducting tests on various workflows that incorporate iterations over dynamic values, the solution appears to be effective. The proposed code changes are currently under review.

It would be great if you would like to give it a try in the PR branch and share your experience with us.

Steps for testing the code changes in the pull request branch:

1. Clone the repository: git clone https://github.com/turbot/steampipe-plugin-github.git
2. Navigate into the repository directory: cd steampipe-plugin-github
3. Switch to the branch issue-388: git checkout issue-388
4. Execute the make command.
5. Execute the query.

Thank you for your patience!

[gabrielsoltz]

Hi @ParthaI, this is working 👌 THANK YOU. I am looking forward to the release.