Automatically deploy the latest LFortran version from the latest master

[certik]

No description provided.

[Shaikh-Ubaid]

Please, could someone possibly share if the build lfortran.wasm and lfortran.js are to be pushed directly into the master branch of this repository or are they to be uploaded at some place?

[certik]

Good question. Given that lfortran.wasm is about 2.5MB, I think we should only push it into the gh-pages branch, but not master, otherwise this repository's size will grow very quickly.

Also, we need to change quite a few things in the pages too, such as the LFortran's master git commit hash, etc.

I think when LFortran's master CI runs, it will ping this repository to rebuild --- not sure what the best approach is, whether with some webhook, or rather LFortran's CI would push the new commit hash into the master of this repository.

This will trigger this repository's CI. This CI will either download the lfortran.wasm, or build it. Either way, it will push the new lfortran.wasm into the gh-pages branch, together with the new website.

So I am thinking here is a TODO list, let me know what you think:

• 1. Create a file here that will store the git commit hash of the LFortran's master commit that is being built
• 2. when we build the website from master, we also check out LFortran and checkout the given commit, and build it into wasm; push everything into gh-pages
• 3. we update LFortran's CI to update the commit hash in the master of this repository, which will trigger an update per previous point
• 4. later, we can investigate if LFortran's CI can upload the lfortran.wasm somewhere (which it also has to build anyway, to check that everything works), and then we can just download it in step 2 instead of building from source.

[Shaikh-Ubaid]

I will share my thoughts on the above points in some time (We currently use nextjs in our frontend and I would probably need to see what could/would work/be-possible with nextjs).

We have a basic/preliminary CI at lfortran/lfortran#157, which I am hoping would/might work for the moment. Currently, I am focusing on the issues https://gitlab.com/lfortran/lfortran/-/issues/773 and https://gitlab.com/lfortran/lfortran/-/issues/771.

[certik]

As of now, the automatic update works, here is an example commit: 8deb3da, but as can be seen there, it puts the LFortran binary directly to master, so the master will grow very quickly (#12).

I think we need some solution to this, either what I described above, or to push the lfortran.wasm into some separate repository, and we download it from there.

[Shaikh-Ubaid]

Yes, got it. I am currently noting my thoughts. I will be sharing them very soon (10-15 mins).

[Shaikh-Ubaid]

I looked into the points that you shared. I think they are great.

Following are my thoughts:

• I think the part building lfortran to wasm should be done at the lfortran repository. Reasons/Benefits:
  • There is a (slight) possibility that a PR submitted might not allow lfortran to build towasm. For example: if there is change related to the cmake variable HAVE_BUILD_TO_WASM, maybe it got removed, etc . Or other example: the facing of the static_assert() error when the build type is Release, discussed at Zulip: Building LFortran in emscripten (this is still to be fixed, the current way is to comment out those specific code lines)
  • So, if a PR affects lfortran's ability to compile to wasm that would be caught right at lfortran's CI and thus the PR can be avoided/delayed/fixed/merged depending on the case.
• I tried building the nextjs site without the presence of the lfortran.js and lfortran.wasm and it builds! So, I guess/think we can keep/host the lfortran.wasm and lfortran.js at some other place.
  • One of the ideas that come to my mind is to rename the tarballs repository to tarballs and binary and maybe we can push the generated lfortran.wasm and lfortran.js in a separate branch in this tarballs and binary repository. (I am unsure how much this is feasible. Please share your views regarding this repository renaming and pushing to a branch in it.)
• From the 1st point (of this comment), if the CI builds successfully, then the CI can upload to appropriate place (related-to/discussed-in 2nd point of this comment). After upload, the CI needs to (somehow) inform the lcompilers_frontend CI about the latest commit.
  • I guess this can be done as you suggested, the lfortran's CI pushing the latest commit into lcompilers_frontend master, then the lcompilers_frontend master fetching the required lfortran.wasm and lfortran.js and then building the site.

[certik]

Yes, I agree with this. LFortran's CI has to build lfortran.wasm no matter what, so that we can ensure no PR ever breaks it. Then we might as well copy the binary somewhere, and tell lcompilers_frontend to rebuild the page with the new binary.

[Shaikh-Ubaid]

I guess this can be done as you suggested, the lfortran's CI pushing the latest commit into lcompilers_frontend master, then the lcompilers_frontend master fetching the required lfortran.wasm and lfortran.js and then building the site.

One of the concerns that I have is that the automatic commits by the lfortran's CI into the master of lcompilers_frontend do not seem/look very great/nice. For example, the below image:

I guess, the master branch of this lcompilers_frontend repository should possibly/hopefully hold the changes/commits (made by developers) related to the code changes so that later, it is hopefully easier to debug if anything breaks. I am unsure, but, probably automated commits and developer commits together might not make up a good commit history.

I just thought of sharing my concern. Anything is fine/works for me.

[certik]

Yes, I don't like it either.

At GitLab I solve this exact issue by using webhooks that trigger the GitLab pipeline that just reruns. So I still trigger the webpage rebuild, but I do not have automatic commits in master. Is there some way to do this using GitHub?

[Shaikh-Ubaid]

Is there some way to do this using GitHub?

I am unsure. I will explore about it.

[certik]

Looks like it can be done: https://kontent.ai/blog/how-to-trigger-github-action-using-webhook-with-no-code/

  # Allows external webhook trigger
  repository_dispatch:
    types:
      - webhook

[czgdp1807]

A quick thought. Is there any feature to choose from a number of recent latest commits? Say latest 3 commits of lfortran and the last stable release. Not very urgent but I think it would be good to have this facility just in case someone wants to resolve to previous versions/commits. May not be a reasonable idea but having options as a user helps sometimes.

[certik]

I was thinking having dev.lfortran.org to just use the latest master.

Then have another domain, perhaps live.lfortran.org that would only server the releases. That way the live.lfortran.org would always work, and dev.lfortran.org might potentially fail if we screw up the master, but we can use it to simply check that the master actually works.

[Shaikh-Ubaid]

From #3 (comment)

Given that lfortran.wasm is about 2.5MB, I think we should only push it into the gh-pages branch, but not master, otherwise this repository's size will grow very quickly.

doubt: As the gh-pages branch is also seems to be a part of this repository, please, could someone possibly share/clarify if pushing to it would increase the respository size?

[Shaikh-Ubaid]

If pushing to the gh-pages might not increase the repository size, please, could someone possibly share if the CI at lfortran can upload the generated lfortran.wasm and lfortran.js to a new/separate branch (let's say wasm_binary) in this current repository (lcompilers_frontend).

[Shaikh-Ubaid]

If pushing to the gh-pages might not increase the repository size, please, could someone possibly share if the CI at lfortran can upload the generated lfortran.wasm and lfortran.js to a new/separate branch (let's say wasm_binary) in this current repository (lcompilers_frontend).

Also, please, could someone possibly share if it would be possible to store lfortran.wasm and lfortran.js for every merge commit in master (I guess, this is similar to what we do for tarballs).

[Shaikh-Ubaid]

Is there any feature to choose from a number of recent latest commits? Say latest 3 commits of lfortran

I am trying for this. Hopefully, it seems it could-be/is possible to support all commits (all commits since the time we incorporate this feature).

[certik]

The tarballs approach allows all commits, currently still visible at: https://lfortran.org/download/, hundreds of versions.

[Shaikh-Ubaid]

The current approach of manually updating the lfortran version in build.sh looks good to me.

Is this issue safe to close?

[certik]

Yes, the current approach is a good one.