Bitbucket Server Catalog Plugin Event Support

[davelil4]

Bitbucket Server Catalog Plugin Event Support

I added event support for the Bitbucket Server catalog backend plugin, as well as the event backend plugin that is for that event support. I follow much of the same logic as the Bitbucket Server Cloud plugin with its event support. If I need to be more rigorous or clear with my testing please let me know.

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or updated documentation
• Tests for new functionality and regression tests for bug fixes
• All your commits have a Signed-off-by line in the message. (more info)

[backstage-goalie]

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/plugin-catalog-backend-module-bitbucket-server	plugins/catalog-backend-module-bitbucket-server	minor	v0.2.2-next.0
@backstage/plugin-events-backend-module-bitbucket-server	plugins/events-backend-module-bitbucket-server	minor	v0.1.0

[github-actions]

Uffizzi Ephemeral Environment Deploying

☁️ https://app.uffizzi.com/github.com/backstage/backstage/pull/19633

⚙️ Updating now by workflow run 6502333863.

What is Uffizzi? Learn more!

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

Hi, would you please be re-open the PR? I still wish to get this merged.

[bforbis]

@davelil4 I am beginning to test this PR now, as I'm very much interested in using this. A quick clarification as I begin testing, I ended up following the README instructions to install the plugins (copied the src as local internals for now), and I followed the events backend instructions to create packages/backend/src/plugins/events.ts and also wire it up to catalog.ts and index.ts.

I then copied a sample paload from atlassian documentation for the repo:refs_changed event and put that in postman

curl request

curl --location 'http://localhost:7007/api/events/http/bitbucketServer' \
--header 'X-Event-Key: repo:refs_changed' \
--header 'Content-Type: application/json' \
--data-raw '{
    "eventKey": "repo:refs_changed",
    "date": "2023-01-13T22:26:25+1100",
    "actor": {
        "name": "admin",
        "emailAddress": "admin@example.com",
        "active": true,
        "displayName": "Administrator",
        "id": 2,
        "slug": "admin",
        "type": "NORMAL"
    },
    "repository": {
        "slug": "rep_1",
        "id": 1,
        "name": "rep_1",
        "hierarchyId": "af05451fc6eb4bf4e0bd",
        "scmId": "git",
        "state": "AVAILABLE",
        "statusMessage": "Available",
        "forkable": true,
        "project": {
            "key": "PROJECT_1",
            "id": 1,
            "name": "Project 1",
            "description": "PROJECT_1",
            "public": false,
            "type": "NORMAL"
        },
        "public": false,
        "archived": false
    },
    "changes": [
        {
            "ref": {
                "id": "refs/heads/master",
                "displayId": "master",
                "type": "BRANCH"
            },
            "refId": "refs/heads/master",
            "fromHash": "197a3e0d2f9a2b3ed1c4fe5923d5dd701bee9fdd",
            "toHash": "a00945762949b7b787ecabc388c0e20b1b85f0b4",
            "type": "UPDATE"
        }
    ],
    "commits": [
        {
            "id": "a00945762949b7b787ecabc388c0e20b1b85f0b4",
            "displayId": "a0094576294",
            "author": {
                "name": "Administrator",
                "emailAddress": "admin@example.com"
            },
            "authorTimestamp": 1673403328000,
            "committer": {
                "name": "Administrator",
                "emailAddress": "admin@example.com"
            },
            "committerTimestamp": 1673403328000,
            "message": "My commit message",
            "parents": [
                {
                    "id": "197a3e0d2f9a2b3ed1c4fe5923d5dd701bee9fdd",
                    "displayId": "197a3e0d2f9"
                }
            ]
        }
    ],
    "toCommit": {
        "id": "a00945762949b7b787ecabc388c0e20b1b85f0b4",
        "displayId": "a0094576294",
        "author": {
            "name": "Administrator",
            "emailAddress": "admin@example.com"
        },
        "authorTimestamp": 1673403328000,
        "committer": {
            "name": "Administrator",
            "emailAddress": "admin@example.com"
        },
        "committerTimestamp": 1673403328000,
        "message": "My commit message",
        "parents": [
            {
                "id": "197a3e0d2f9a2b3ed1c4fe5923d5dd701bee9fdd",
                "displayId": "197a3e0d2f9",
                "author": {
                    "name": "Administrator",
                    "emailAddress": "admin@example.com"
                },
                "authorTimestamp": 1673403292000,
                "committer": {
                    "name": "Administrator",
                    "emailAddress": "admin@example.com"
                },
                "committerTimestamp": 1673403292000,
                "message": "My commit message",
                "parents": [
                    {
                        "id": "f870ce6bf6fe633e1a2bbe655970bde25535669f",
                        "displayId": "f870ce6bf6f"
                    }
                ]
            }
        ]
    }
}'

When posting this from Postman, I get a successfull response from Postman but then the backstage processor fails with the following message:

2023-09-06T19:24:08.359Z backstage error Subscriber "BitbucketServerEntityProvider" failed to process event Cannot read properties of undefined (reading 'self') type=plugin stack=TypeError: Cannot read properties of undefined (reading 'self')
[1]     at BitbucketServerEntityProvider.getLocationEntity (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:262:45)
[1]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[1]     at async BitbucketServerEntityProvider.onRepoPush (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:297:25)
[1]     at async BitbucketServerEntityProvider.onEvent (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:278:13)
at async /Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:34:11
[1]     at async Promise.all (index 0)
[1]     at async DefaultEventBroker.publish (/Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:31:5)

Is this error expected? The error message doesn't seem to be too helpful, but I'm guessing it's possible this is because of the dummy data in the payload? I'm still working through some networking permissions to actually fully test this with bitbucket.

[bforbis]

After fussing with some WAF settings I was able to get this working in a full end-to-end test and happy to say it accepted the event for me with no errors! So this seems to work fine with legit payloads from bitbucket.

[davelil4]

I'm not sure why these other checks are failing but I adjusted the docs if you want to check @bforbis .

[bforbis]

@davelil4 I am beginning to test this PR now, as I'm very much interested in using this. A quick clarification as I begin testing, I ended up following the README instructions to install the plugins (copied the src as local internals for now), and I followed the events backend instructions to create packages/backend/src/plugins/events.ts and also wire it up to catalog.ts and index.ts.

I then copied a sample paload from atlassian documentation for the repo:refs_changed event and put that in postman

curl request
When posting this from Postman, I get a successfull response from Postman but then the backstage processor fails with the following message:

2023-09-06T19:24:08.359Z backstage error Subscriber "BitbucketServerEntityProvider" failed to process event Cannot read properties of undefined (reading 'self') type=plugin stack=TypeError: Cannot read properties of undefined (reading 'self')
[1]     at BitbucketServerEntityProvider.getLocationEntity (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:262:45)
[1]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[1]     at async BitbucketServerEntityProvider.onRepoPush (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:297:25)
[1]     at async BitbucketServerEntityProvider.onEvent (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:278:13)
at async /Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:34:11
[1]     at async Promise.all (index 0)
[1]     at async DefaultEventBroker.publish (/Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:31:5)

Is this error expected? The error message doesn't seem to be too helpful, but I'm guessing it's possible this is because of the dummy data in the payload? I'm still working through some networking permissions to actually fully test this with bitbucket.

@davelil4 any idea if the error message here could be cleaned up to be more human readable?

[davelil4]

@davelil4 I am beginning to test this PR now, as I'm very much interested in using this. A quick clarification as I begin testing, I ended up following the README instructions to install the plugins (copied the src as local internals for now), and I followed the events backend instructions to create packages/backend/src/plugins/events.ts and also wire it up to catalog.ts and index.ts.
I then copied a sample paload from atlassian documentation for the repo:refs_changed event and put that in postman
curl request
When posting this from Postman, I get a successfull response from Postman but then the backstage processor fails with the following message:

2023-09-06T19:24:08.359Z backstage error Subscriber "BitbucketServerEntityProvider" failed to process event Cannot read properties of undefined (reading 'self') type=plugin stack=TypeError: Cannot read properties of undefined (reading 'self')
[1]     at BitbucketServerEntityProvider.getLocationEntity (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:262:45)
[1]     at processTicksAndRejections (node:internal/process/task_queues:96:5)
[1]     at async BitbucketServerEntityProvider.onRepoPush (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:297:25)
[1]     at async BitbucketServerEntityProvider.onEvent (webpack-internal:///../../plugins/catalog-backend-module-bitbucket-server/src/providers/BitbucketServerEntityProvider.ts:278:13)
at async /Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:34:11
[1]     at async Promise.all (index 0)
[1]     at async DefaultEventBroker.publish (/Users/bforbis/athena/git/bforbis/athena-backstage/node_modules/@backstage/plugin-events-backend/dist/cjs/HttpPostIngressEventPublisher-6e928c08.cjs.js:31:5)

Is this error expected? The error message doesn't seem to be too helpful, but I'm guessing it's possible this is because of the dummy data in the payload? I'm still working through some networking permissions to actually fully test this with bitbucket.

@davelil4 any idea if the error message here could be cleaned up to be more human readable?

@bforbis I'll look into soon as I get the chance. Just to speed up time, what are the "WAF" settings you mentioned that you changed to fix the error? I had never previously gotten this error so it would help me out to know.

[bforbis]

@bforbis I'll look into soon as I get the chance. Just to speed up time, what are the "WAF" settings you mentioned that you changed to fix the error? I had never previously gotten this error so it would help me out to know.

Ignore the part about the WAF. That was just something blocking me from testing end-to-end with Bitbucket directly (ie: I needed to update my firewall to allow bitbucket to send requests to backstage).

The error that I saw before was from me testing the event webhook URL manually using Postman. Since I didn't yet have the bitbucket connection setup, I tried testing manually by using the sample payload provided on Bitbucket's doc site.

I am not surprised that this caused an error within backstage, as that sample payload is all dummy data. However, the error message reported by Backstage was very confusing and did not actually clue me into what was going wrong. I would have expected some sort of message like "Could not find associated entity", but instead we got some crazy JS undefined behavior error.

[freben]

Sure.

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[awanlin]

Hi @davelil4, this has 3 approvals, is there anything blocking this from being merged? Personally this seems good to 🚢

@Rugvip, look like for some reason there's an open change you requested but it looks resolved 🤔

That leads me here: #19633 (review)

[davelil4]

Hi @awanlin last I checked I was still looking at getting some requested changes resolved. If that got resolved, I will check my computer when I get home and getting the changes merged.

[awanlin]

Thanks for the update @davelil4, we are getting close to a year for this PR and there is a TON of value in here that IMO we should get out into the world. We can always iterate and make it better in smaller PRs from there.

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity from the author. It will be closed if no further activity occurs. If the PR was closed and you want it re-opened, let us know and we'll re-open the PR so that you can continue the contribution!

[davelil4]

Bumping to unmark stale

[awanlin]

Hi @davelil4, were you able to follow up? We should really try and get this merged, I can't see any reason not too!

[davelil4]

Hi @davelil4, were you able to follow up? We should really try and get this merged, I can't see any reason not too!

@awanlin i will rebase master and take a look later today if possible. If I can't get to it this week I will definitely take a look this weekend and get it in a state to merge.

[davelil4]

Bumping to unmark stale, I will fix conflicts later.

[davelil4]

@Rugvip can you please check the requested changes when you get the chance.

[davelil4]

@awanlin Any chance you can help with getting these requested changes reviewed? If possible, thank you.