Git integration (formerly Spock)

[jesseleite]

Hey all!

We are currently looking at git integration in v3, and are looking for community feedback around Spock. What were your pain points with Spock in v2, and what kinds of features/functionality would you like to see in v3?

Also, Spock was mainly used for running git commands, but technically it could listen to any events and run any commands. If you were using Spock for more than just committing files, we'd love to know how you were using it 🙂

[DavidStrada]

Ability to push to a different repo, e.g. Statamic-content, so we can keep separate repos. code / content.

[ehibun]

A big part of my use case would be to invalidate caches such as CDNs when content is updated. In v3 specifically, I planned to use Spock to trigger the static site generator and kick off the deployment process to get the results of that uploaded to a CDN, to separate the control panel from the site proper.

[fspoettel]

Some things that could be improved:

• I tried to log Spock errors when pushing since those would indicate that my "source of truth" repo is out-of-date. That was hard and the logger picked up a lot of things that weren't errors
• Without the queue driver, Spock slows down the user interface to a point where I sometimes thought that an error had occurred. Especially while syncing image uploads to the repo or git-lfs

[jesseleite]

That was fast! Good thoughts so far 👍

A big part of my use case would be to invalidate caches such as CDNs when content is updated.

@peterhicks Couldn't you already do this with Spock in v2? Spock allowed you to add any cli commands to Spock's command runner (ie. cache clear commands, etc). Was something missing for your use case?

Without the queue driver, Spock slows down the user interface to a point where I sometimes thought that an error had occurred. Especially while syncing image uploads to the repo or git-lfs

@fspoettel Totally agree with this, though that's why we implemented the queue driver option. Unfortunately, I don't think we can default to this because: A) Laravel apps use sync/file queue driver by default, and B) We can't assume redis/etc is installed and working. That said, we could clear up documentation around recommending the use of a proper queue driver 👍

[lachr]

When creating many entries (fetching data from an api) it creates for each Entry a commit. It would be cool if this type of changes could be made in just one commit.

[simonhamp]

I’ve actually been toying with the idea of building an add-on for this that has a better UI.

It would be truly great to be able to see a commit history and review changes, potentially even with the ability to flag stuff to be passed back to the user who edited to fix. Also a button to be able to Push changes manually back to the remote.

[curtismchale]

My biggest issue was with updating the site from the Admin UI which would trigger spock and then deploy from Forge back to the site causing a period I had to wait as the deploy happened. During this window I couldn't update content because it wouldn't work.

[jesseleite]

Again great feedback all.

My biggest issue was with updating the site from the Admin UI which would trigger spock and then deploy from Forge back to the site

Yeah @curtismchale, for those circumstances, it's probably recommended to disable quick-deploy in Forge. Not sure there's a way around that, unless Forge has some sort of API webhook that would allow us to push back without triggering an auto-deploy.

[curtismchale]

@jesseleite I wonder if you could set a time interval to wait on a push? So you'd commit the changes but only push after XX minutes to the repository? Even set it to 0 and leave it up to Devs to customize the value if they need to.

[jesseleite]

@curtismchale totally. We're already exploring options for when/how it can be triggered 👍 My point was that if you have quick deploy enabled in Forge, you're going to have a circular loop of automatic events that I don't think we can do much about (I don't think).

[elliottregan]

Better support or instructions for automated deployments and CI/CD pipelines. We've been setting up automatic deployments on more projects. Spock has caused a bit of friction in our workflows for two reasons:

1. When Spock pushes new commits to origin, we don't want to kick off a new build and deploy of the site. That's easy to solve with path filters in our build tool, but it is worth noting because of the extra considerations we need to take when including Statamic in a repo. There are a few other suggestions we considered, but I'm not sure if we landing on something that made everyone happy. Different hosting providers might have more limited options than what we were working with.
2. During a deploy, there is a period of a few minutes while new changes on the server have the chance of being overwritten. If a deploy is triggered and the new artifact is being built, a new change made on the server will be erased by the new build. It might still be stored in git and pushed to origin, but because of issue 1, that commit won't trigger a new deploy, and the content editor will thing their work is lost. Worse, nobody will know the new content is gone right away, the editor edits the files on the server again, which has become out of sync with what's on origin, and now we have a merge conflict.
   We got around this instituting a "content freeze" during deploys, but that requires communication, and still doesn't prevent a CEO from going in to change a field without anybody's knowledge.

Basically, all of the issues we ran into while getting Spock set up were solvable, but we aren't sure if we chose the best solution, and we aren't certain any solution we looked at will work with every hosting situation. Any solution seemed to involve some additional rules or imposed structure. Whether that was just keeping good communication between devs and editors, enforcing a particular git branch strategy, or something else, there wasn't "one solution" that could be repeated for each project.

Since it is "just git", I understand that it will be hard to solve these issues in a way that works for everyone. Having some additional documentation or a suggested configuration that works for different common scenarios would be helpful. Even a list a other plugins to safely force a content freeze, or handle merge conflicts from the Admin UI would take some of the worry away.

[mikemartin]

We’re using SemaphoreCI to deploy to my clients load-balanced servers. We added a simple [BOT] flag to Spock’s commit message so we can suppress a deployment being triggered for content changes.

Spock command:

environments:
  - staging
commands:
  - "git add -A"
  - "git commit -m '[BOT] {{ url }} updated by {{ committer:username }}'"
  - "git push"

Forge deploy script for each staging server:

git fetch
git reset --hard origin/master
./bin/deploy.sh &

Our shared deploy script in our repo:

npm install && \
npm run prod && \
php please search:update && \
php please clear:cache && \
git rev-parse HEAD > site/themes/example/templates/api/version.html

SemaphoreCI script:

if ( git show $REVISION | grep -e '\[BOT\]' ); then echo 'Bot deploy handled automatically in statamic'; else curl -X POST $STATAMIC_STATAMIC && sleep 60 && ( curl https://example.com/api/version | grep "$REVISION" ); fi

[1stevengrant]

I've definitely seen some weird caching issues with V2. The client would make some changes, Spock would do its thing and then the homepage would 404 until I blew the cache.

Ultimately, to resolve we've gone the route of implementing a CMS server for the control panel and then doing a deploy and using the file cache.

I actually think this is a good workflow overall for most of my clients but it would be nice not to have the caching issue.

[jesseleite]

I've definitely seen some weird caching issues with V2. The client would make some changes, Spock would do its thing and then the homepage would 404 until I blew the cache.

@1stevengrant interesting. I'm curious what about Spock's git add/commit/push would be causing the caching issue? Do you have the same caching issues when not using Spock?

PS. If it is related to Spock, a workaround in v2 (not necessarily an ideal solution) could be to add php please clear:cache to Spock's commands_after array. For now anyway.

[1stevengrant]

yeah, it may be a total coincidence. I haven't seen it first hand but https://stuffandnonsense.co.uk does every so often when he deploys and he doesn't use Spock. I've never had the time to deep dive into what might be causing it.

…

On Thu, 4 Jun 2020 at 15:21, Jesse Leite ***@***.***> wrote: I've definitely seen some weird caching issues with V2. The client would make some changes, Spock would do its thing and then the homepage would 404 until I blew the cache. @1stevengrant <https://github.com/1stevengrant> interesting. I'm curious what about Spock's git add/commit/push would be causing the caching issue? Do you have the same caching issues when not using Spock? PS. A workaround in v2 (not necessarily an ideal solution) could be to add php please clear:cache to Spock's commands_after array. For now anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#199 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3OWIZE7OJYX5YFCEJWL3RU6UYLANCNFSM4NRZVJ4Q> .

-- Steven Grant

[jasonvarga]

Do you have both clear:cache and clear:stache in your deployment script? I've seen it happen there. You only need the one.

[jasonvarga]

@mikemartin Great tip on the condition. We could put together steps to do that in Forge etc and would definitely be helpful. No need to redeploy when changing content on production.

[jesseleite]

Oh right, guess I was wrong. The Forge deploy script could check last commit for [BOT] and exit early or wrap the deploy logic in a condition. Good call.

[1stevengrant]

Yes! Will give that a whirl

…

On Thu, 4 Jun 2020 at 16:08, Jason Varga ***@***.***> wrote: Do you have both clear:cache and clear:stache in your deployment script? I've seen it happen there. You only need the one. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#199 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA3OWOPA3M3LPZQJVAWQPLRU62FBANCNFSM4NRZVJ4Q> .

-- Steven Grant

[austenc]

Used spock to keep client content updates in sync so all I had to do from the development side was a git pull to get their latest changes when I needed to make an update to a site.

Because SSH'ing in to the server to make a commit was a hassle, spock made it easy :)

Although sometimes I had to SSH in anyway when a conflicting change or something would break forge auto deployments.