Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Tugboat.qa for PR sandboxes #4351

Closed
ghost opened this issue Mar 25, 2020 · 50 comments · Fixed by backdrop/backdrop#3105
Closed

Use Tugboat.qa for PR sandboxes #4351

ghost opened this issue Mar 25, 2020 · 50 comments · Fixed by backdrop/backdrop#3105

Comments

@ghost
Copy link

ghost commented Mar 25, 2020

Description of the need
We are considering using Tugboat.qa for PR sandboxes for Backdrop core (since that's Tugboat's advertised feature).

Proposed solution
Backdrop core will have a new, hidden directory with Tugboat-specific files inside. These will trigger preview sites to be built whenever a PR is opened. Previews will be automatically rebuilt when the PR is updated (e.g. with new commits). Comments will be posted to the PR when preview sites (sandboxes) are ready with login details so people can test the new functionality/bug fix.

Alternatives that have been considered
We are currently using Zen.ci for all of this, but there have been some issues and so we'd like to see if Tugboat is a better fit...

How to test this functionality:

  1. Fork BWPanda's Backdrop repo: https://github.com/BWPanda/backdrop (pretend this is the main Backdrop repo)
  2. Clone and checkout the tugboat branch (pretend this is Backdrop's 1.x branch)
  3. Create a new branch (for the changes you want to make to Backdrop)
  4. Make, commit and push your changes
  5. Create a PR against BWPanda:tugboat (pretend you're making it against backdrop:1.x)
  6. Wait for the Tugboat preview to be generated and displayed in the PR, then test it to see if your changes are visible
  7. Report back here with your experience

This issue has been given the 'bug' milestone candidate label as it's hoped this can get into the next bugfix release. This exception is due to the fact that the proposed additional code won't affect Backdrop sites at all, only the repository/PRs (i.e. no chance of introducing bugs to existing Backdrop sites).


Advocate: @BWPanda

Status: Code has been merged, now we just need to setup Tugboat, etc. (see #4351 (comment)).

@ghost
Copy link
Author

ghost commented Mar 25, 2020

PR here: backdrop/backdrop#3105

I don't think we'll be able to test it until we merge it and setup Tugboat properly, but I've been testing it in my own repo here: https://github.com/BWPanda/backdrop/pull/1

The first few comments were me getting the GitHub API working, then I finally got it posting a custom comment to GitHub when the sandbox was ready (Tugboat can post a comment itself, but it can't be customised and we need the login details added there, so custom comments needed). Then it was posting a new comment every time the PR was updated, even though the URL, username and password hadn't changed. So I then made it delete old comments before posting a new one (but only deleting comments where the URL was the same - because I closed and re-opened the PR at one point, the URL changed, so that's why no comments above that point were deleted).

All in all this seems to be working nicely!

@ghost
Copy link
Author

ghost commented Mar 25, 2020

If/when we merge this, we'll need to do the following things to get it working properly:

  • Setup a new Tugboat project
    • Ask Tugboat team nicely to upgrade free plan for us
    • Add @quicksketch, @BWPanda (and anyone else?) as users
  • Add backdrop/backdrop repo to Tugboat project
    • Enable following settings for repo in Tugboat:
      • Set Pull Request Deployment Status
      • Build Previews for Forked Pull Requests
      • Rebuild Orphaned Previews Automatically
      • Rebuild Stale Previews Automatically
    • Add an environment variable called BACKDROP_GITHUB_TOKEN with a GitHub API key/token
    • Set the Backdrop CI user as the one who'll be posting comments
    • Build a preview for the 1.x branch and set it as the base preview

@ghost
Copy link
Author

ghost commented Mar 26, 2020

Actually, others can test this. Pretend my tugboat branch is Backdrop's 1.x and make a PR against it in my repo. E.g. https://github.com/BWPanda/backdrop/pull/2

@ghost
Copy link
Author

ghost commented Mar 26, 2020

One thing we could do is to alter the comment posted to GitHub saying that Tugboat has a built a sandbox site for testing the PR, it's a new thing we're trying, and that people should test it out if possible and let us know how it goes. We can do this alongside the existing Zen.ci sandboxes.

@docwilmot
Copy link
Contributor

Tried that, think it didnt work? Or must I wait?

@ghost
Copy link
Author

ghost commented Mar 26, 2020

@docwilmot https://github.com/BWPanda/backdrop/pull/3 Didn't work because it was for the wrong branch (but you obviously realised that as you also did https://github.com/BWPanda/backdrop/pull/4 which didn't work (I think) because you need to create a branch from mine first, make changes, then do the PR (same way you'd do with Backdrop 1.x). I suspect it didn't work because your changes don't include the .tugboat directory... Maybe try adding that and committing to see if that triggers it...?

@ghost
Copy link
Author

ghost commented Mar 26, 2020

Hmmm... Maybe it's a permissions thing...?

@ghost
Copy link
Author

ghost commented Mar 31, 2020

Copying comment from PR 4:

If I look at my PR that worked (https://github.com/BWPanda/backdrop/pull/2/files) I can't see any changes to the .tugboat directory, just the single change I made. In your PR I can see the addition of the Tugboat files, which make me think something weird is going on there...

Can you try a new branch/PR with a simpler change that originates from my branch?

@jenlampton
Copy link
Member

@BWPanda can you update the parent issue with more on the motivation to switch? I think ZenCI has been working really well for us, and I'm not compelled by "there have been some issues". I think if there were a clearer statement of why that would help!

@ghost
Copy link
Author

ghost commented Apr 17, 2020

@jenlampton I updated the OP with a link to a specific issue that @klonos has been experiencing on and off for a few years. As for other reasons, here's part of the discussion that prompted this issue:

@klonos (15/03/2020):

This question goes to @quicksketch and the rest of our core committers: why not use tugboat instead of ZenCI for our core PRs? (assuming that we can get a free plan)
...I have noticed that tugboat sandboxes take 3+min to be generated, but perhaps we can apply the same trick of pre-built images to spin them up faster(??).
...there are some known issues with ZenCI (sandboxes not generated, sandboxes generated point to install.php, links to tests not working sometimes, etc.), and it kinda worries me is that there is only a single person behind ZenCI (@Gormartsen).

@quicksketch (21/03/2020):

Tugboat is specifically made to provide PR sandboxes, that's it's advertised feature. What we're doing now with Tugboat demos is sort of atypical (though Simplytest.me now uses Tugboat too, so maybe not). Anyway, I'd be down with using Tugboat for PR sandboxes too, though Zen.ci seems to work most of the time so I haven't really given it a priority. Tugboat seems a bit more supported than Zen.CI from my perspective though. When something goes wrong with Zen we're entirely dependent upon @Gormartsen (though he has been reliable and great). Tugboat seems to have a few more contact points.

@BWPanda (21/03/2020):

I've had good success with Tugboat's Slack channel for support. I'd be happy to look into this if you like...

In this week's dev meeting @quicksketch brought up this issue but mistakenly said I came up with this idea. I was just going off the discussion between @klonos and @quicksketch (who I thought was for this, but in the meeting he seemed to suggest he's on the fence about it...).

Hope that helps clear things up a bit.

@klonos
Copy link
Member

klonos commented Apr 17, 2020

...I'm not compelled by "there have been some issues".

...there are some known issues with ZenCI (sandboxes not generated, sandboxes generated point to install.php, links to tests not working sometimes, etc.)

@jenlampton "links to tests not working sometimes" is basically backdrop-ops/backdropcms.org#276

As for the other two problems I've mentioned, well funnily enough, I managed to reproduce both of them in a single issue 😅 ...I was about to ask you and @stpaultim for a quick review of #3906 ...so I closed both PRs linked from that issue in order to refresh the sandboxes. I waited ~3min and then re-opened both PRs (both simple 2-liner changes + 2 new png files added; so pretty simple PRs)

Screen Shot 2020-04-17 at 6 56 36 pm

🤷

[edit] the php7 and php5 tests in backdrop/backdrop#2766 have both failed with random failures, so I've logged in ZenCI to trigger the tests again. The tests seem to be now stuck in this loop no matter how many times I restart them (that issue with appending -1 and then -2 in the URL). What usually fixes this is to close/wait/reopen yet another time 🤷 🤷

@klonos
Copy link
Member

klonos commented Apr 17, 2020

...I should probably mention that all these issues are solved if I close the PR, wait 2-3min, and then re-open the PR. Often closing and reopening does not work the first time, so I have to repeat the close/wait/reopen ritual a second time. If it fails again, then that's when I usually ping @Gormartsen on Gitter about it, and he does his magic to fix things 🙏

@klonos
Copy link
Member

klonos commented May 17, 2020

It took 4-5 times of closing/reopening the PR (after previously trying to restart tests - which didn't work): backdrop/backdrop#2929

😓 😓 😓 😓 😓

@ghost
Copy link
Author

ghost commented May 19, 2020

Related issue @jenlampton opened for doing the same thing for the B.org, API and Forum repos too: backdrop-ops/backdropcms.org#621

@ghost
Copy link
Author

ghost commented Sep 5, 2020

Here's another test I did: https://github.com/BWPanda/backdrop/pull/5

Here are the steps for others to test this as well:

  1. Fork my Backdrop repo: https://github.com/BWPanda/backdrop (pretend this is the main Backdrop repo)
  2. Clone and checkout the tugboat branch (pretend this is Backdrop's 1.x branch)
  3. Create a new branch (for the changes you want to make to Backdrop)
  4. Make, commit and push your changes
  5. Create a PR against BWPanda:tugboat (pretend you're making it against backdrop:1.x)
  6. Wait for the Tugboat preview to be generated and then test it to see if your changes are visible

I'll update the OP with these too. Also, I'm advocating for this issue now.

@ghost ghost self-assigned this Sep 5, 2020
@ghost ghost added this to the 1.18.0 milestone Sep 5, 2020
@herbdool
Copy link

herbdool commented Sep 5, 2020

I think it's looking good. Perhaps it could go in a bug fix release since it's tooling and not a new feature?

Wondering about the password. Is it possible to get a random one each time? It's not totally secure but could lower risk of them being taken over. Not sure how much of a risk that is.

@jenlampton
Copy link
Member

jenlampton commented Sep 11, 2020

FWIW I agree with @herbdool and think I'd rather have the sandbox refreshed with every PR.

We need this in order to start using it.

Well, yes :) But we will also need a tugboat account before we can start using it. It's a chicken and an egg thing I suppose, but what if we can't get the kind of account we need? We haven't even confirmed that getting what we need is possible yet. It seems like committing the tool before we even know if it's possible to use it might be jumping the gun a little. I suppose we can always take it out again if that happens (but fingers crossed that it won't!)

@ghost
Copy link
Author

ghost commented Sep 11, 2020

Well I tried making the previews stay intact between commits, but initial attempts failed, so refreshing them fully between commits would be easier...

@jenlampton I see your point, however it seems to me that leaving the code there for now is easiest. If we revert the change while waiting to see what happens with a free account upgrade, then:

  • If we get the upgrade, we have to add the code back in later (initial commit, revert, then commit again = 3 changes)
  • If we don't get the upgrade, we can always revert later (initial commit, revert later = 2 changes)

In other words, since the code's already in place, let's leave it there and see what happens.

@ghost
Copy link
Author

ghost commented Sep 11, 2020

So, current status is that we're waiting on @quicksketch to create a new Tugboat project. Once he's done that and added me as an admin to it, I can do the rest of the tasks (if you like) including sending an email to Tugboat asking for a free upgrade.

@klonos If/once this is all working, we can revisit the issue of rebuilding after each commit after we've seen it in action and get some feedback from users.

@jenlampton
Copy link
Member

jenlampton commented Sep 11, 2020 via email

@klonos
Copy link
Member

klonos commented Sep 11, 2020

@herbdool and @jenlampton I see your point; there's indeed value in starting from scratch each time we need to test a code change, but my goal would be to lower the barrier for people to contribute by testing. Consider this:

  1. dev files a PR that fixes an issue or implements a new feature, and requests review/testing
  2. non-dev person uses the PR sandbox to test and sets up things (user accounts, nodes, views, menus, config changes)
  3. non-dev person provides feedback about a very minor thing (say a comma missing, a typo, or a string tweak)
  4. dev amends their code and pushes changes

If step 2 requires a couple of mins to set up, the tester is likely to repeat and test again. If it takes say 15min or more to reconfigure, I doubt that they would as easily be motivated to start from scratch 🤷

If we need to hard-reset the PR sandbox, we have the option to close the PR -> wait a couple of mins for the sandbox to be properly destroyed (that's with ZenCi - not sure if required with tugboat) -> reopen PR. In other words, that's basically 2 easy steps that take us a click -> 2min wait -> another click. People testing need to rebuild their entire setup to re-test, which takes a considerable amount of time, and I'm afraid that this will decrease the motivation of testers to contribute.

@klonos If/once this is all working, we can revisit the issue of rebuilding after each commit after we've seen it in action and get some feedback from users.

Agreed 👍

@jenlampton
Copy link
Member

jenlampton commented Sep 11, 2020 via email

@herbdool
Copy link

@jenlampton exactly. On a recent PR I caused confusion with testers because I pushed a commit on a PR but didn't fully refresh because I forgot there was a crucial config change.

@jenlampton jenlampton modified the milestones: 1.16.3, 1.17.1 Sep 15, 2020
@jenlampton jenlampton modified the milestones: 1.17.1, 1.17.2 Sep 30, 2020
@jlfranklin
Copy link
Member

I'm sorry I didn't notice this issue sooner. I don't think this should have been included for several reasons:

  1. As @jenlampton points out, it's not used anywhere. I see there is already a backdrop-ops/backdrop-tugboat repository that can be used to test this, and @BWPanda mentions other repos he's been using for testing. When the project has an upgraded Tugboat account, and we're ready to run with it, the Tugboat account can be repointed to the main repo. However...

  2. Tugboat is not part of the project, it's part of the infrastructure. Travis CI didn't need to include a .travis-ci.yml file or any related scripts. They were added during the test box setup process as a response to a new or updated PR via a GitHub webhook. Is there a way to keep the .tugboat config external to the main repository?

  3. We've avoided adding platform or tool-specific items. The .editorconfig file was accepted, because it's an open standard widely used by editors and IDEs. The .lando config was not, because it was for a specific tool. There are Platform.sh images and Docker images created by the project, yet there is nothing in the main repository that defines those images. They're built externally, as part of the release process. Why is Tugboat special?

  4. Is there anything here that a contributing developer can or should change? As a contributing developer, would this suggest that I need to get a Tugboat account to work on this project?

  5. Travis CI has been very generous to the project, and @Gormartsen's support is unanimously held in high regard. Is there some way we can give back to Travis CI by helping them debug the few issues we've encountered? No platform is perfect, and I can't help thinking we're just trading known issues in Travis for unknown issues in Tugboat.

This is the first commit I'm going to have to revert in Silkscreen. Silkscreen doesn't use Tugboat, nor do I have any intention of doing so. And I don't want these configs to show up in the tarballs and zip files that Github automatically creates.

@ghost
Copy link
Author

ghost commented Oct 4, 2020

I see there is already a backdrop-ops/backdrop-tugboat repository that can be used to test this

Not exactly. That repo is used for spinning up demo sites for users to try Backdrop in general: https://backdropcms.org/demo The Tugboat files in the core repo are for spinning up copies of Backdrop specifically for previewing PRs.

Travis CI didn't need to include a .travis-ci.yml file or any related scripts.

Actually it did (#2051), we just don't use Travis anymore: #1906 and #1947 We use Zen.ci instead, and it does have files/scripts in core too: https://github.com/backdrop/backdrop/tree/1.x/core/misc/zen-ci

Why is Tugboat special?

As far as I'm aware, Tugboat requires configuration files to be added to the repo where PRs are being created into order to work its magic and spin up sites for previewing those PRs. I'm happy to be corrected if I'm wrong about this: https://docs.tugboat.qa/

As a contributing developer, would this suggest that I need to get a Tugboat account to work on this project?

I don't think so. Just as the Zen.ci files in core shouldn't indicate that. We can add a README to the .tugboat directory if you think that'll help clarify things...

Is there some way we can give back to Travis Zen CI by helping them debug the few issues we've encountered?

You're welcome to get @Gormartsen's attention and point him to the issues we need advice/help on: backdrop-ops/backdropcms.org#276 backdrop-ops/backdropcms.org#551 #3364

@ghost ghost removed the pr - needs work label Oct 9, 2020
@ghost
Copy link
Author

ghost commented Oct 15, 2020

Just thought I'd mention here that some new contributors are apparently of the opinion that Zen.ci fails all the time, that's how bad it's getting:

Are you talking about the automated zenci/test/php53 and zenci/test/php70? My impression has been that those always fail but it seems like it's for unrelated reasons.

(#4686 (comment))

And an update: @quicksketch emailed Tugboat asking about a new, free account for PRs, and we should be hearing back from them about that tomorrow.

@quicksketch
Copy link
Member

Here's the full email from CEO Matt Westgate:

Thanks for reaching out and being patient with me in my response. Looping in our head of biz-dev, Jaden as well.

Happy to continue supporting Backdrop and the Dragon Drop Dragon! The next tier beyond 40GB of storage bumps you over into the Enterprise tiers. The good news that you'd get a dedicated instance and plenty of space... 200GB of storage, I believe. The one concern I have though is financial, as Tugboat incurs a monthly financial cost for the hardware to run it. That cost would be just shy of $200/month.

Are you open to having a conversation about how Tugboat is positioned from a marketing perspective in our Backdrop collaboration? I'd love to collaborate and see how we can find a balance to the spend, support the Backdrop community, and create value for each other. I feel like there's a lot of potentials we haven't tapped into, but I also know you have other sponsors we also need to be aware of.

I'd love to hear your thoughts on what that might look like! Do you feel like you could starting brainstorming on this? I feel like you all have a better idea of what's more realistic and in line with your brand and values from a marketing perspective.

I think what's being implied here is that Tugboat might provide us a higher tier account if we provide them more marketing opportunities. I'm not sure what options we have, since we're already promoting Tugboat for demos and in the README.md. We don't have any substantial GitHub integration on BackdropCMS.org so the per-PR sandboxes are going to be fairly hidden away. But it's definitely worth having the conversation. I'll reach back out and we can have a phone conversation with them.

@olafgrabienski
Copy link

@quicksketch Do you have an idea what kind of marketing opportunities are of interest for them?

@klonos
Copy link
Member

klonos commented Oct 26, 2020

We could discuss this during the next outreach or dev meeting, but some possibilities could be:

  • Automatic comments in the respective issue, each time a Tugboat demo is being created from a PR (issues are visible by more people than PRs are)
  • Some prominent add space in our forum/api/main sites.

I have always wanted our sites to remain add-free for ever, but this is in order for us to get a much needed, otherwise paid-for service in exchange. I think that we should make an exception.

@ghost
Copy link
Author

ghost commented Nov 4, 2020

I've updated the checklist above. I've done everything I can, just need @quicksketch to get me a GitHub API key for the core repo to add to the Tugboat account so it can post comments to PRs. Then just waiting on Tugboat to increase the account limit and I think we're done.

@jenlampton
Copy link
Member

Then just waiting on Tugboat to increase the account limit and I think we're done.

The new account is supposed to reduce the need for an account limit increase. @BWPanda I think you should have admin access to the new (second) Tugboat account.

just need @quicksketch to get me a GitHub API key for the core repo to add to the Tugboat account so it can post comments to PRs

I just logged in and can see that @dragonbot is already authorized to post comments on PRs.

Is there anything else you need?

@ghost
Copy link
Author

ghost commented Nov 4, 2020

@jenlampton I see the account is 40GB now. Last I looked it was only 2GB. That's what I meant.

We still need to add a GitHub API let as an environment variable to Tugboat. This is because I disabled the default posting of comments and did it a different way. So our custom script needs the API key...

@jenlampton
Copy link
Member

jenlampton commented Nov 4, 2020 via email

@ghost
Copy link
Author

ghost commented Nov 4, 2020

@jenlampton I found Dragonbot's login in 1Password and was able to do it myself.

@ghost
Copy link
Author

ghost commented Nov 4, 2020

This is all done now and working! Thanks to everyone who helped! I'm sure there'll be things we can fix/tweak in the future, but we can open new issues for them. Closing now.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants