ARROW-1299: [Docs] Publish nightly docs build from crossbow by jorisvandenbossche · Pull Request #173 · apache/arrow-site

jorisvandenbossche · 2021-12-15T12:18:55Z

No description provided.

github-actions · 2021-12-15T12:19:12Z

https://issues.apache.org/jira/browse/ARROW-1299

jorisvandenbossche

@kszucs @kou what do you think of this approach?

I wanted to used https://github.com/dsaltares/fetch-gh-release-asset to download the docs from the crossbow release, but since INFRA doesn't allow such third-party actions, I copied that in for now.

jorisvandenbossche · 2021-12-15T15:33:24Z

+            git add docs/dev --all
+            git commit -m "Updating dev docs (build ${DATE})"
+            echo "git push"
+            git push


I think this push is not yet working because that is disallowed by default from pushes (to avoid an infinite loop of actions re-triggering themselves), according to https://stackoverflow.com/a/58393457/653364 (which suggests you need to use a custom personal access token instead of the default one).
I can maybe also use a similar approach as in deploy.yml

I think that the information is old. It will work.

I have a job that pushes a commit to gh-pages for all push: https://github.com/ruby/csv/blob/master/.github/workflows/test.yml#L96-L135
And it works.

OK, then this should probably work indeed (once it is merged and running on master?)

jorisvandenbossche · 2021-12-15T15:36:14Z

+          cd asf-site
+          echo "$(git log -1)"
+          git config user.name "$(git log -1 --pretty=format:%an)"
+          git config user.email "$(git log -1 --pretty=format:%ae)"


This sets the commit author to the last author on the asf-site branch (so a random committer). Do we care about who exactly is the commit author for this? (it will be the last one of us that committed something to the site)

I first did

git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com"

to have the commit author be a more anonymous bot, but then that user does not have push rights? (or is there a way to workaround this?)

We can use the following:

git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com"

@jorisvandenbossche Could you try this?

I already did I think? (unless I am misunderstanding)

kszucs · 2021-12-15T18:31:18Z

@jorisvandenbossche thanks for working on this!

How about replacing the fetch release asset with an archery subcommand? It whould be fairly easy using either github3.py or PyGithub.

kou

Concern: apache/arrow-site repository size will be increased steadily. We may need to remove the previous commit to update dev docs (only if the previous commit is an update dev docs commit) before we update dev docs.

kou · 2021-12-15T21:20:59Z

+          git config user.name "$(git log -1 --pretty=format:%an)"
+          git config user.email "$(git log -1 --pretty=format:%ae)"
+          mkdir -p docs/dev
+          tar -xvzf ../docs.tar.gz -C docs/dev --strip-components=1


Could you replace docs/dev/c_glib/index.html with docs/c_glib/index.html after tar -x?
We want to use https://github.com/apache/arrow-site/blob/master/_docs/c_glib/index.md for c_glib/index.html.

See also: https://github.com/apache/arrow/blob/master/dev/release/post-09-docs.sh#L59

kou · 2021-12-15T21:21:37Z

+  push:
+  pull_request:


Do we need these events?

No, that's just temporary for testing on this PR. After that only the cron job will be sufficient

OK.
Could you remove push and pull_request before we merge this?

We can keep pull_request if we use the forked repository's gh-pages branch instead of apache/arrow-site's asf-site branch like our existing .github/workflows/deploy.yml does.

Yes, will certainly update this before merging

We can keep pull_request if we use the forked repository's gh-pages branch instead of apache/arrow-site's asf-site branch like our existing .github/workflows/deploy.yml does.

OK, I see how this is done in deploy.yml. Now, for the docs I think that's probably less relevant, as it is not content from a PR that is being built (as is the case for the main site here), but rather just fetched from elsewhere. But we can always add it later if it would be useful to have.

kou · 2021-12-15T21:24:47Z

+          cd asf-site
+          echo "$(git log -1)"
+          git config user.name "$(git log -1 --pretty=format:%an)"
+          git config user.email "$(git log -1 --pretty=format:%ae)"


We can use the following:

git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com"

kou · 2021-12-15T21:35:49Z

+            git add docs/dev --all
+            git commit -m "Updating dev docs (build ${DATE})"
+            echo "git push"
+            git push


I think that the information is old. It will work.

I have a job that pushes a commit to gh-pages for all push: https://github.com/ruby/csv/blob/master/.github/workflows/test.yml#L96-L135
And it works.

jorisvandenbossche · 2021-12-16T14:20:49Z

How about replacing the fetch release asset with an archery subcommand? It whould be fairly easy using either github3.py or PyGithub.

I thought you were suggesting to implement this in archery, but it seems it actually already exists! (the archery crossbow download-artifacts and latest-prefix options, testing them out now)

jorisvandenbossche · 2021-12-16T17:42:47Z

Downloading the artifacts with archery is working now! (much nicer)
Querying the latest available nightly build number with archery will require apache/arrow#11978

kszucs · 2021-12-16T20:56:49Z

I thought you were suggesting to implement this in archery, but it seems it actually already exists!

I were suggesting that, I forgot that we have both upload and download artifacts already implemented :D

kou · 2021-12-17T21:33:18Z

Concern: apache/arrow-site repository size will be increased steadily. We may need to remove the previous commit to update dev docs (only if the previous commit is an update dev docs commit) before we update dev docs.

@jorisvandenbossche @kszucs What do you think about this?

jorisvandenbossche · 2021-12-23T15:32:49Z

Concern: apache/arrow-site repository size will be increased steadily. We may need to remove the previous commit to update dev docs (only if the previous commit is an update dev docs commit) before we update dev docs.

Yes, this is indeed a problem (I raised a similar concern when adding the multiple versions of the docs, as that also steadily increases the repo size). For the dev docs specifically, we of course only need to have the latest version and can thus overwrite / clean-up the git history to avoid increasing the repo size.

The options I was thinking about:

In the action, remove the history of just the dev docs directory with something like https://github.com/newren/git-filter-repo/. This rewrites history and thus requires force pushing from inside the action.
In the action, we could also completely re-initialize the asf-site branch from the current content + updated dev docs (in the idea that we mostly care about history in the mater branch, and less so in the asf-site branch?). But similarly as above, this requires force pushing from inside the action (so probably not much advantage for this option compared to the one above)
Keep the action as is (additional commit, so no need to force push), but from time to time clean-up the asf-site branch and force push manually (we could have a similar script to remove the history in the dev docs directory)

Your idea of removing the previous commit to update dev docs (only if the previous commit is an update dev docs commit) could also be an option, but that will miss some of those commits, and also requires force pushing.

I was a bit hesitant to take any option that requires a force push (so didn't yet add anything in this PR, leaving the manual clean-up from time to time), but maybe that's not actually a problem?

jorisvandenbossche · 2022-01-11T10:27:29Z

I updated this to now also use archery to get the tag of the latest doc build, which seems to work.

Any more thoughts about the comment above (how to handle the increasing size / is it OK to force push from the action?)

Also, would you be fine with going forward with this PR, and give this a try in practice? We can still update it to handle git branch cleaning to reduce history size later in a follow-up, I think.

kszucs · 2022-01-11T10:58:12Z

I updated this to now also use archery to get the tag of the latest doc build, which seems to work.

Any more thoughts about the comment above (how to handle the increasing size / is it OK to force push from the action?)

I think force pushing would be sufficient.

Also, would you be fine with going forward with this PR, and give this a try in practice? We can still update it to handle git branch cleaning to reduce history size later in a follow-up, I think.

Agree, though the build is actually failing due to missing permissions.

jorisvandenbossche · 2022-01-11T11:01:44Z

though the build is actually failing due to missing permissions.

According to the comment of @kou above (#173 (comment)), this should work (once merged in master?)
(but honestly no idea if it will actually work)

kou

though the build is actually failing due to missing permissions.

According to the comment of @kou above (#173 (comment)), this should work (once merged in master?) (but honestly no idea if it will actually work)

Yes. Pull request should not change the original repository for security reason.

It's not failed on the fork repository:

Also, would you be fine with going forward with this PR, and give this a try in practice? We can still update it to handle git branch cleaning to reduce history size later in a follow-up, I think.

OK.

kou · 2022-01-11T21:06:27Z

+
+      - name: Fetch Crossbow branches
+        run: |
+          cd crossbow
+          git fetch origin +refs/heads/*:refs/remotes/origin/*


Can we use fetch-depth: 0 here?

https://github.com/actions/checkout/#Fetch-all-history-for-all-tags-and-branches

Suggested change

- name: Fetch Crossbow branches

run: |

cd crossbow

git fetch origin +refs/heads/*:refs/remotes/origin/*

fetch-depth: 0

kou · 2022-01-11T21:10:18Z

+  push:
+  pull_request:


OK.
Could you remove push and pull_request before we merge this?

We can keep pull_request if we use the forked repository's gh-pages branch instead of apache/arrow-site's asf-site branch like our existing .github/workflows/deploy.yml does.

jorisvandenbossche · 2022-01-12T14:16:33Z

Fixed a few last things (the date in the commit message, also ensured we don't commit the ignored .doctrees), verified that this still works on my fork, and then as last commit removed the push and pull_request triggers.

With that, I am going to merge this, so we can see how this goes in master and further improve if needed.
Thanks both for the feedback!

One thing I already noticed is that, depending on the exact time of the day, it can happen that the latest nightly build tag is already found but still building, and so the download doesn't do anything. If that turns out to be a problem, we could change the cron job to run twice a day, so that at least one of both will download the latest docs.

ARROW-1299: [Docs] Publish nightly docs build from crossbow

7520533

Temp: also trigger on pull request

7fd5a0a

jorisvandenbossche force-pushed the ARROW-1299-host-dev-docs branch from 277da62 to b85c4da Compare December 15, 2021 12:35

test with local copy of action

55bdd8f

jorisvandenbossche force-pushed the ARROW-1299-host-dev-docs branch from b85c4da to 55bdd8f Compare December 15, 2021 12:43

jorisvandenbossche added 5 commits December 15, 2021 13:53

fetch origin

c5bc473

add git config user

6064f52

use actions/checkout for checking out asf-site and pushing

3c23a28

fetch asf-site after downloading docs

bdb791b

try fix path

ace7576

jorisvandenbossche force-pushed the ARROW-1299-host-dev-docs branch from 569fc77 to 6ec620f Compare December 15, 2021 14:48

more path handling

065d8d7

jorisvandenbossche force-pushed the ARROW-1299-host-dev-docs branch 2 times, most recently from 589b928 to 2196d2d Compare December 15, 2021 15:04

use author of last commit

3e73c1b

jorisvandenbossche force-pushed the ARROW-1299-host-dev-docs branch from 2196d2d to 3e73c1b Compare December 15, 2021 15:07

jorisvandenbossche added 2 commits December 15, 2021 16:22

add debug log

081a12c

add license and readme of copied actions

c2051b4

jorisvandenbossche requested review from kou and kszucs December 15, 2021 15:27

jorisvandenbossche commented Dec 15, 2021

View reviewed changes

kou reviewed Dec 15, 2021

View reviewed changes

try use archery to download artifacts

9ad965c

jorisvandenbossche force-pushed the ARROW-1299-host-dev-docs branch 2 times, most recently from b5fb4dd to b336bc0 Compare December 16, 2021 15:23

specify arrow path

caaa0e8

jorisvandenbossche force-pushed the ARROW-1299-host-dev-docs branch from b336bc0 to caaa0e8 Compare December 16, 2021 15:26

jorisvandenbossche added 6 commits December 16, 2021 16:31

also checkout crossbow

c4750b6

no need to specifu remote queue

7f19db2

don't fetch (we already fetched latest)

15f378a

fetch branches manually

e264469

remove fetch-gh-release-asset

d50bdad

copy c_glib index.html + change back github user

37326eb

kou reviewed Dec 17, 2021

View reviewed changes

Comment thread .github/workflows/devdocs.yml Outdated

Use archery to get latest doc build tag

a1885cf

kou reviewed Jan 11, 2022

View reviewed changes

kou and others added 5 commits January 12, 2022 06:46

Simplify GitHub Actions bot email

8b28728

Fix DATE and use fetch-depth 0

2df86d2

remove push+pull_request

4967c50

don't add with --all to respect ignored files

9af54f9

remove push+pull_request once more

3aea038

jorisvandenbossche merged commit 6b5d7b9 into apache:master Jan 12, 2022

jorisvandenbossche deleted the ARROW-1299-host-dev-docs branch January 12, 2022 14:16

jorisvandenbossche mentioned this pull request Jan 12, 2022

ARROW-1299: [Docs] Allow trigger nightly devdocs job manually #176

Merged

Conversation

jorisvandenbossche commented Dec 15, 2021

Uh oh!

github-actions Bot commented Dec 15, 2021

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kszucs commented Dec 15, 2021

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche commented Dec 16, 2021

Uh oh!

jorisvandenbossche commented Dec 16, 2021

Uh oh!

kszucs commented Dec 16, 2021

Uh oh!

Uh oh!

kou commented Dec 17, 2021

Uh oh!

jorisvandenbossche commented Dec 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorisvandenbossche commented Jan 11, 2022

Uh oh!

kszucs commented Jan 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jorisvandenbossche commented Jan 11, 2022

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jorisvandenbossche commented Jan 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

jorisvandenbossche commented Dec 23, 2021 •

edited

Loading

kszucs commented Jan 11, 2022 •

edited

Loading