Set up actions/workflow for web hooks from other openMINDS repositories

[ehennestad]

Set up separate workflows for web hooks to the following repositories:

openMINDS_Documentation
openMINDS_Instances

Each action should ideally be triggered by a push or PR to one of these repositories.

@olinux

[ehennestad]

Added workflow for running test if updates are made to openMINDS_Documentation:
https://github.com/openMetadataInitiative/openMINDS_MATLAB/blob/pipeline/.github/workflows/test_doclinks.yml

This should be triggered on a push or PR to the pipeline branch of openMINDS_Documentation.

I'll tag you as well @lzehl

[ehennestad]

As far as I understand, we would have to add a new workflow to the openMINDS_Documentation repository:

trigger_target_repositories.yml

name: Trigger target repositories

on:
  # Triggers the workflow on push or pull request events for the "pipeline" branch
  push:
    branches: [ "pipeline" ]
  pull_request:
    branches: [ "pipeline" ]

jobs:
  trigger:
    runs-on: ubuntu-latest
    steps:
    - name: Trigger target repositories
      run: |
        curl -XPOST -u "${{secrets.PAT_USERNAME}}:${{secrets.PAT_TOKEN}}" -H "Accept: application/vnd.github.everest-preview+json" -H "Content-Type: application/json" https://api.github.com/repos/openMetadataInitiative/openMINDS_MATLAB/actions/workflows/test_doclinks.yml/dispatches --data '{"ref": "pipeline"}'

[olinux]

Hi @ehennestad ,
Here's the way the build pipelines work:

We have 3 different types of repositories:

• "Modules" (e.g. openMINDS submodules such as "core", "SANDS", but also "instances"
• "Central" the repository at https://github.com/openMetadataInitiative/openMINDS
• "Targets" the target repositories such as "openMINDS_Python", "openMINDS_Java", "openMINDS_MATLAB" but also "openMINDS_documentation"

The idea is that the "modules" contain their individual items (such as schemes and instances). Once they have a new commit, they trigger the build process of "central" which aggregates the information across all of those modules and merges them in a single (versioned) view. Once this is done, the "target" build pipelines are triggered to then consume the centrally provided information to build their "language-specific" products.

The pipeline trigger for MATLAB (the "pipeline" branch) is already configured in the central openMINDS repository - which means that it will be executed automatically whenever there is any change in central (and therefore indirectly from any "module" such as "instances"). Please do not rely on "documentation" as a source since this is a "target" module -> You should be able to find the same information in openMINDS central which is supposed to be the only source of truth of any "target" module. If you're lacking some specific information you can only find in documentation, we should check how we can move it to the central repository.

[lzehl]

@olinux I fully agree. With one addition: we are not going to move documentation content to the central repository (with potential exceptions directly related to the metadata models/schemas)

[ehennestad]

@olinux The specific issue I have is that in the openMINDS_MATLAB package I generate URLs to the pages describing schemas on the ReadTheDocs documentation. Furthermore, these URLs are used in object displays within MATLAB, and it has happened on several occasions that the way a URL is generated have changed due to changes in the openMINDS_documentation pipeline and the effect is that these links in MATLAB are broken and its very hard to detect. That's why I would like to have a small test suite to test these generated URLs when changes are made specifically on openMINDS_documentation's pipeline branch.

That is why, as I sketched out in the example above, I need a web hook for when changes have been made to the pipeline branch of openMINDS_documentation.

[ehennestad]

The same is true for the instances "module", where I have some dependencies that rely not on the instances themselves, but how the instances are organised by the build script on the pipeline branch

EDIT: I see that the instances library does not have a pipeline.

[lzehl]

@ehennestad the instance an openMINDS product.

For building pipelines of any non-module/central repo:

• ground truth schemas: central repo
• ground truth instances: instances repo

[ehennestad]

Some more insights on the openMINDS_instances:

Changes on openMINDS_instances do trigger the build action of openMINDS/pipeline .
Changes on openMINDS_instances do not cause any changes to the openMINDS repo when the central openMINDS pipeline is run, so nothing is pushed to openMINDS/main
Therefore the openMINDS_upstream action on openMINDS/main is not invoked and none of the target repositories are triggered.

I don't know if this is intended or not, but at the moment, if the openMINDS_instances is changed, none of the target repositories are triggered, but they should be in my opinion.

[ehennestad]

Let's recapitulate:

1. I need to trigger one workflow in openMINDS_MATLAB/pipeline when pushes/PRs are made to openMINDS_documentation as described above. From what I understand, we need to add a workflow (as I outlined above) in openMINDS_documentation/pipeline that triggers openMINDS_MATLAB/pipeline. So, If I understand this correctly(?), can I proceed to make a PR on openMINDS_documentation/pipeline with such a workflow?

2. I need to trigger one workflow in openMINDS_MATLAB/pipeline when pushes/PRs are made to openMINDS_instances. Based on the previous discussions here, this should already happen

The pipeline trigger for MATLAB (the "pipeline" branch) is already configured in the central openMINDS repository - which means that it will be executed automatically whenever there is any change in central (and therefore indirectly from any "module" such as "instances").

However in practice this does not happen (as I described in the previous post), and I am not sure if this is intentional or not

[lzehl]

@ehennestad to clarify openMINDS_instances will never trigger updates to openMINDS (main). The only thing planned is that openMINDS_instances will receive a workflow/action that verifies / validates instances according to their referenced schemas (not implemented yet).

Targets that make use of instances (which are not all) should be triggered by updates in instances (e.g. documentation).

Targets as MATLAB and Python could also purely provide functions to interactively pull instances on demand (TBD). If they should be always auto-triggered with a change in instances I agree that this needs to be set up (if this was not yet done).

[ehennestad]

Currently openMINDS_instances triggers the central openMINDS build pipeline.

If this will never have an effect in practice (i.e. not trigger updates on openMINDS/main, then I suggest that trigger I removed and replaced with one that triggers target repositories directly.

Targets as MATLAB and Python could also purely provide functions to interactively pull instances on demand

This is currently an option in openMINDS_MATLAB, but it breaks when there are structural changes to openMINDS_instances, like this, so It would be nice to have workflows that can do sanity checks.

[lzehl]

@ehennestad but the structural change should not have an effect on your functions to be honest. then you should reconsider your code.

Also why do you think a change in openMINDS_instances triggers the central openMINDS pipeline? This should not be the case and if this is true it is a mistake we have to fix

[lzehl]

For instances or schema subdirectories: avoid any code dependencies because they may change. Until now they basically do not have any meaning for schemas or instances but to make it easier for us developers to find stuff. we could also fully remove subdirectories

[ehennestad]

but the structural change should not have an effect on your functions to be honest. then you should reconsider your code.

In order to find instance files, you need to build their path. You can of course scan the whole folder structure but that is less efficient.

Also why do you think a change in openMINDS_instances triggers the central openMINDS pipeline? This should not be the case and if this is true it is a mistake we have to fix

This:
https://github.com/openMetadataInitiative/openMINDS_instances/blob/4089d81b734afbf9dbe0a57c16d21017f438b9f2/.github/workflows/openMINDS_upstream.yml#L13

[lzehl]

@ehennestad about the subdirectories: up to you but there is definitely no guarantee for those to be constant. you really should deal with this in the code differently for the schemas. For the instances we can discuss for at least some assumptions to be true. Let's discuss this in a meeting though.

@olinux could you double check/explain why there is an openMINDS_upstream.yml pointing from instances to central/main?

[olinux]

@lzehl I've checked - and I think the reason for having an openMINDS_upstream.yml is kind-of a left-over: We have discussed once to have instances as part of the central repository as well at some point (remember) and additionally, I think we've followed the approach that all "submodule" repositories have the same build pipeline. @ehennestad is right - either we keep the instances repo independent and extend it with a trigger to the target repositories as well or we actually let the build process integrate the instances documents in the central repository as well to make sure the trigger is pushed and the submodules are notified. I kind of like the latter idea because it would mean that there truly is only a single source of truth for all target repos - and that is the central openMINDS repo. Downside of it is that we would have a lot of duplication (although we would know where the source of truth is).

@ehennestad : I also think it's more safe to indeed go through all instances regardless of their folder structure - it's something you do at build time, so the minor performance penalty you get shouldn't be problematic.

[ehennestad]

@ehennestad about the subdirectories: up to you but there is definitely no guarantee for those to be constant. you really should deal with this in the code differently for the schemas. For the instances we can discuss for at least some assumptions to be true. Let's discuss this in a meeting though.

I understand, and that's exactly why I opened this issue, so the responsibility falls on me to write the necessary tests to detect if something changes.

@ehennestad : I also think it's more safe to indeed go through all instances regardless of their folder structure - it's something you do at build time, so the minor performance penalty you get shouldn't be problematic.

At the moment I don't include the instances in the openMINDS_MATLAB package. There is an option to download individual instances on demand directly from openMINDS_instances, and in this case it seems more efficient to build the path/url for the jsonld file in the repository.

Another reason I thought to build the path/url is that I will not make a new version release of openMINDS_MATLAB for every change on the central repository. People that are using a versioned release (and not pulling the repo) could the still get access to the latest instances if instances can be retrieved from openMINDS_instances on demand