Add CI system and integrate into pages #5

bytesnake · 2021-03-11T08:19:20Z

We should build and publish the book automatically to rust-ml.github.io/book on each push:

add small CI system which builds the book
add gh branch with artifacts

The text was updated successfully, but these errors were encountered:

liufuyang · 2021-03-17T20:47:59Z

I think we can use this https://github.com/peaceiris/actions-mdbook

I used it earlier in my personal mdbook note and it seems working pretty well. Should I create a PR here for you?

It's basically adding a yml file then change the repo setting like this:

Or... we are not using a standard mdbook perhaps it needs to be tweaked or? 🤔

bytesnake · 2021-03-18T08:27:36Z

We're using mdbook here, but with some tweaks 😆 so there is the plugin https://github.com/rust-ml/book/tree/master/mdbook-scientific which basically adds special syntax for latex/gnuplot code blocks. The environment would need:

mdbook
a compiled version of the mdbook-scientific plugin
LaTeX, perhaps gnuplot
bib2xhtml

I know that these are a lot of dependencies, but perhaps you can use the mdbook action as baseline and then install/compile the rest

liufuyang · 2021-03-18T11:25:34Z

I see. I am not that experienced with github action yet but this is probably can be done.

liufuyang · 2021-03-23T11:30:12Z

@bytesnake I tried to make an PR #8 to make the first version CI work. How do you like the general idea? It is not fully working yet, but I guess with bib2xhtml install then we will be closer to it. It would be nice if you can help make a master-test-build branch for me to test the CI build later on?

liufuyang · 2021-04-19T12:19:07Z

#8 indeed can trigger github actions but the build failed. I will take a deeper look later this week :)

liufuyang · 2021-04-19T18:02:03Z

So I tried to let the build scrip just run twice then it can be built. Seems like a race condition thing perhaps? So when building the first time, some build process didn't wait for the needed middle files to be generated on disk?
You can see the error message like this:

Run mdbook build || mdbook build
2021-04-19 17:44:06 [INFO] (mdbook::book): Book building has started
Error in chapter 1. InvalidMath("! I can\'t find file `e9ea986c569b3a41be529169.tex\'.", "", 18446744073709551615)
Error: -19 17:44:07 [ERROR] (mdbook::utils): Error: The preprocessor exited unsuccessfully
2021-04-19 17:44:07 [INFO] (mdbook::book): Book building has started
2021-04-19 17:44:07 [INFO] (mdbook::book): Running the html backend

I noticed second build can work is because I remembered days ago I made the image to work with the book on that build branch. As I am on a new computer now, I noticed the image was not working, but simply try running again without touching anything made the build works.

But I guess it still only works with the book of the version that is on the build branch. Need to solve how to make master branch also works later.

Anyway you have something here already https://rust-ml.github.io/book/

liufuyang · 2021-04-19T18:23:22Z

Just some helpful note here for anyone would like to try to use the image on master to build the book and see the error:
Firstly checkout master branch.
Then update book.toml setting with bib2xhtml = "/github/bib2xhtml/"
Then you can run docker build

docker run --rm -v $(pwd):/github/workspace liufuyang/mdbook-scientific:0.3.7 build

or if you want to take a look inside docker:

docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it liufuyang/mdbook-scientific:0.3.7

# then you can run: mdbook build

The error looks like this:

2021-04-19 18:22:56 [INFO] (mdbook::book): Book building has started
Block empty, but file `hyperplane` was not found!
Block empty, but file `hyperplane` was not found!
Error in chapter 1.: InvalidReference("could not find reference to `hyperplane` in line 5")
2021-04-19 18:22:56 [ERROR] (mdbook::utils): Error: The preprocessor exited unsuccessfully

I have no clue what hyperplane is about...

bytesnake · 2021-04-26T15:08:26Z

first: thanks for all the work! I will try to reproduce the build error inside the docker image. I have currently too many side-projects to really focus on one thing, what always works is 1-to-1 meetings and debugging code together. Perhaps I'll find time this evening, but otherwise I'll propose that you join in the bi-monthly meetings and we can look into the issue afterwards?

liufuyang · 2021-04-26T17:54:54Z

Oh cool. I might have time tonight as well. I haven't joined the bi-monthly meetings before but I would like to come join once and listen to your discussion. However I might be difficult to find time on Wednesday as I have some club activity on Wednesday evening. But let's see, maybe I can join someday.

bytesnake · 2021-04-26T19:39:56Z

I'm giving it a try now, posted a link in zulip.

However I might be difficult to find time on Wednesday as I have some club activity on Wednesday evening. But let's see, maybe I can join someday.

would be awesome!

liufuyang · 2021-04-26T21:12:57Z

Okay, so I made a large (if not huge) image to include all the rust build environment as well.
You can see the code changes here that for building this large image.

So basically you can either try pull the image (which might take a long time)

docker pull liufuyang/mdbook-scientific:0.3.7-large

or build the image on your own (also very slow the first time) by checking out that master-test-build-large-image branch and run build script.

* If you just pull the image (the easy way)

Then I guess it's simple as you can test your Rust code changes by running the image via interaction mode, such as:

docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it liufuyang/mdbook-scientific:0.3.7-large

Then you can do build and run as if you are inside an ubuntu:

cd /github/workspace/mdbook-scientific
cargo install --path .
cd /github/workspace/
mdbook build

Then your updated Rust code will run

* If you decide to build the image on your own (the harder way)

Basically you can just cp your original Dockerfile into a backup file, then use the Dockerfile from the branched I mentioned above.
Then build the image on your own

docker build -t test:latest .

Each time you update the Rust code, you need to re-run the build step above so the new binary is baked into the image.
Then you can run it as below to build the book (which should execute your updated Rust code!)

docker run --rm -v $(pwd):/github/workspace test:latest build
# or like this to do interaction way:
#  docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it test:latest

@bytesnake Let me know whether this helps :)

liufuyang · 2021-04-26T21:36:43Z

Here I just put some command I ran as a demo:

 docker run --rm -v $(pwd):/github/workspace --entrypoint /bin/bash -it liufuyang/mdbook-scientific:0.3.7-large
root@1dd4dd6cf0a8:/github/workspace# ls
Dockerfile  Dockerfile.back  LICENSE-APACHE-2.0  LICENSE-CC-BY-SA  LICENSE-MIT  README.md  book  book.toml  build.docker.sh  code  fragments  literature.bib  mdbook-scientific  src
root@1dd4dd6cf0a8:/github/workspace# cd mdbook-scientific/
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cargo install --path .
  Installing mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Updating crates.io index
   Compiling version_check v0.9.3
   Compiling cfg-if v1.0.0
   Compiling libc v0.2.94
   Compiling proc-macro2 v1.0.26
   Compiling unicode-xid v0.2.1
   Compiling autocfg v1.0.1
   Compiling syn v1.0.70
   Compiling memchr v2.3.4
   Compiling ryu v1.0.5
   Compiling bitflags v1.2.1
   Compiling proc-macro2 v0.4.30
   Compiling unicode-xid v0.1.0
   Compiling syn v0.15.44
   Compiling getrandom v0.2.2
   Compiling lexical-core v0.7.6
   ...
   Compiling mdbook v0.3.7
   Compiling mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Finished release [optimized] target(s) in 7m 07s
   Replacing /root/.cargo/bin/mdbook-scientific
    Replaced package `mdbook-scientific v0.1.0 (/book/mdbook-scientific)` with `mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)` (executable `mdbook-scientific`)
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cd /github/workspace                  
root@1dd4dd6cf0a8:/github/workspace# mdbook build
2021-04-26 21:27:48 [INFO] (mdbook::book): Book building has started
2021-04-26 21:27:48 [INFO] (mdbook::book): Running the html backend
root@1dd4dd6cf0a8:/github/workspace#

Now you see it compiles (as it is on the build branch).
Then I keep the the container running but in the editor I go to the scientific book main function add a simple println!("my hack..."); as the first line in the main function, then I run the following commands in container as: (following from the terminal prompt above)

root@1dd4dd6cf0a8:/github/workspace# cd mdbook-scientific/
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cargo install --path .
  Installing mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Updating crates.io index
   Compiling mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)
    Finished release [optimized] target(s) in 19.33s
   Replacing /root/.cargo/bin/mdbook-scientific
    Replaced package `mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)` with `mdbook-scientific v0.1.0 (/github/workspace/mdbook-scientific)` (executable `mdbook-scientific`)
root@1dd4dd6cf0a8:/github/workspace/mdbook-scientific# cd ..
root@1dd4dd6cf0a8:/github/workspace# mdbook build
2021-04-26 21:32:22 [INFO] (mdbook::book): Book building has started
my hack...
2021-04-26 21:32:22 [ERROR] (mdbook::utils): Error: Unable to parse the preprocessed book
2021-04-26 21:32:22 [ERROR] (mdbook::utils): 	Caused By: expected value at line 1 column 1
root@1dd4dd6cf0a8:/github/workspace#

You can see the changes in Rust code taking effect. Hopefully this helps! :)
(The simple println seemed broke the mdbook build, but I guess you know why...)

bytesnake · 2021-04-27T20:39:30Z

the asset config was just set to the wrong path. Replacing assets = "mdbook-scientific" by assets = "src/ fixes the error

liufuyang · 2021-04-27T20:48:53Z

aha? let me give it a try the other day :)
Mind if I ask why in the master branch the assets is set as assets = "mdbook-scientific"? Is it for easy development or? Otherwise perhaps we can change it to src?

bytesnake · 2021-04-27T20:55:06Z

I'm not sure but the use of assets is not explained .. @quietlychris

quietlychris · 2021-04-27T21:47:43Z

I honestly am not sure why it got changed from "src" to "mdbook scientific" although looking through my commits, I was the one to change it 😅 I think I may have been playing around with experimenting with different locations for the k_means code prior to that commit and forgot to move it back to the right position. Sorry about that!

bytesnake · 2021-04-27T21:59:22Z

no worries 😄 the only place where the source folder is actually used is when a latex file is loaded. So it could probably be replace, but haven't looked into it yet

liufuyang · 2021-04-30T12:52:30Z

CI should be ready after this #9

quietlychris · 2021-06-02T18:48:36Z

@bytesnake Just wanted to follow up on this, and maybe try to have the CI pushing to the Github Pages site over the next two weeks. I think that as liufuyang mentioned in his last comment in #9, would it possibly make sense to move mdbook-scientific out of this repository and call it as a dependency when needed? At the moment, I don't think that any of the content that I've added is using it as the moment, and I agree that it might make sense to move it into it's own repo like rust-ml/mdbook-scientific so as not to gate this repo on it. If you're onboard, I can try to take care of that tonight. Any thoughts?

bytesnake · 2021-06-03T18:19:01Z

@bytesnake Just wanted to follow up on this, and maybe try to have the CI pushing to the Github Pages site over the next two weeks. I think that as liufuyang mentioned in his last comment in #9, would it possibly make sense to move mdbook-scientific out of this repository and call it as a dependency when needed? At the moment, I don't think that any of the content that I've added is using it as the moment, and I agree that it might make sense to move it into it's own repo like rust-ml/mdbook-scientific so as not to gate this repo on it. If you're onboard, I can try to take care of that tonight. Any thoughts?

yes moving mdbook-scientific to its own repo and making it optional if not used sounds reasonable

quietlychris · 2023-02-04T16:40:05Z

CI updates to Github Actions with mdbook publication for new structure in #12

liufuyang mentioned this issue Mar 23, 2021

Prepare a docker image for build mdbook-scientific #8

Merged

liufuyang mentioned this issue Apr 30, 2021

Add CI to build the book and host via github pages #9

Merged

bytesnake closed this as completed in #9 Jun 3, 2021

bytesnake reopened this Jun 3, 2021

quietlychris closed this as completed Feb 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CI system and integrate into pages #5

Add CI system and integrate into pages #5

bytesnake commented Mar 11, 2021 •

edited by quietlychris

Loading

liufuyang commented Mar 17, 2021 •

edited

Loading

bytesnake commented Mar 18, 2021

liufuyang commented Mar 18, 2021

liufuyang commented Mar 23, 2021

liufuyang commented Apr 19, 2021

liufuyang commented Apr 19, 2021 •

edited

Loading

liufuyang commented Apr 19, 2021 •

edited

Loading

bytesnake commented Apr 26, 2021

liufuyang commented Apr 26, 2021

bytesnake commented Apr 26, 2021

liufuyang commented Apr 26, 2021 •

edited

Loading

liufuyang commented Apr 26, 2021 •

edited

Loading

bytesnake commented Apr 27, 2021

liufuyang commented Apr 27, 2021

bytesnake commented Apr 27, 2021

quietlychris commented Apr 27, 2021

bytesnake commented Apr 27, 2021

liufuyang commented Apr 30, 2021

quietlychris commented Jun 2, 2021

bytesnake commented Jun 3, 2021

quietlychris commented Feb 4, 2023

Add CI system and integrate into pages #5

Add CI system and integrate into pages #5

Comments

bytesnake commented Mar 11, 2021 • edited by quietlychris Loading

liufuyang commented Mar 17, 2021 • edited Loading

bytesnake commented Mar 18, 2021

liufuyang commented Mar 18, 2021

liufuyang commented Mar 23, 2021

liufuyang commented Apr 19, 2021

liufuyang commented Apr 19, 2021 • edited Loading

liufuyang commented Apr 19, 2021 • edited Loading

bytesnake commented Apr 26, 2021

liufuyang commented Apr 26, 2021

bytesnake commented Apr 26, 2021

liufuyang commented Apr 26, 2021 • edited Loading

* If you just pull the image (the easy way)

* If you decide to build the image on your own (the harder way)

liufuyang commented Apr 26, 2021 • edited Loading

bytesnake commented Apr 27, 2021

liufuyang commented Apr 27, 2021

bytesnake commented Apr 27, 2021

quietlychris commented Apr 27, 2021

bytesnake commented Apr 27, 2021

liufuyang commented Apr 30, 2021

quietlychris commented Jun 2, 2021

bytesnake commented Jun 3, 2021

quietlychris commented Feb 4, 2023

bytesnake commented Mar 11, 2021 •

edited by quietlychris

Loading

liufuyang commented Mar 17, 2021 •

edited

Loading

liufuyang commented Apr 19, 2021 •

edited

Loading

liufuyang commented Apr 19, 2021 •

edited

Loading

liufuyang commented Apr 26, 2021 •

edited

Loading

liufuyang commented Apr 26, 2021 •

edited

Loading