Continuous Integration, releases and ready-to-use rolling images

[eine]

• The first part of this issue was implemented in Use docker multistage builds, travis stages, yaml aliases and deploy to dockerhub #489.
• Some of the content was moved to ghdl/docker, sections Use cases for the actually available images, Let's have some fun! [use GHDL live/local/ci demo] and Extending base images are now available at USE_CASES.md. Packing is available at Packing artifacts docker#2.
• Documentation overhaul and Project organization were moved to 1138-4EB/ghdl-io#1. These are being discussed in Landing site for GHDL #506.

---

Table of contents

• Maintainability
• More to come
• Tasks

---

Maintainability

• Use an encrypted travis access token (2013-01-28-token-token-token, encryption-keys, triggering-builds) to cross-trigger builds:
  • Move files corresponding to stage X (let it be 2, 3 or 4) to a separate branch (say, extended). Have travis automatically trigger a build of branch extended every time a build of branch master is successful.
  • Same as above, but host the branch in a different repo.

See 1138-4EB/ghdl-io#1 and #506.

More to come

I believe that having rolling docker images is a major feature for the project. Yet, it is not good enough for my target audience, which are new/young users with little to no contact with digital design. Forcing them to use the shell (say vim, nano, emacs...) is not the best starting point.

ghdl/ext:vunit-gtkwave is the first step to make it easier. The launch script automatically mounts the current path in the container, and the X socket is shared, so that GUI applications can be run (i.e. gtkwave). The user can move and edit files with the tools in the host (file explorer, dolphin, nautilus, gedit, notepad++...). Still, the console is required to run the scripts, but VUnit makes a pretty good job simplifying it.

However, an X server is required in order to run ghdl/ext:vunit-gtkwave. This makes it slightly more difficult to use it on a remote host (say play-with-docker or a machine in the lab). Sure, SSH can be used to forward the required ports, but it requires to set up users and keys in the remote machine.

There are some interesting projects that can be integrated to provide a functional web-based IDE. Indeed, there is a very interesting discussion going on in #111.

I was about to sketch a very basic editor with a file manager and codemirror, when I found eclipse/che. It is a really promising evolution of eclipse/ide. Basically it is a stack composed of multiple orchestrated docker containers with a pretty easy entrypoint: docker run eclipse/che start.

I tried it locally and it is sooooo slow to spin up. It downloaded 10 different images (1.5GB total) to create an empty machine with a blank project. I usually create and remove multiple workspaces every day, and it would make me lose my patience. However, I love the look and feel. Xilinx's SDK is based on Eclipse, so it is quite familiar to users that approach hardware design from an embedded software point of view (I haven't checked if Xilinx projects can actually be used in eclipse/che). Sigasi, which is the reference many of us have in mind when thinking about possible free tools for VHDL design, is also based on Eclipse.

Anyway, the main point is that it works. Just spin it up, create a stack with a single machine based on ghdl/ext:vunit and you are ready to go. Love it:

![ghdl_pwd_demo_eclipseche] There is currently a bug when using eclipse/che in PWD (eclipse/che#3595). I can record a screencast locally, but I'd rather wait to make it clear that there is no 'trick' (i.e. hidden setup or configuration steps).

Note that when the stack is defined, predefined commands can be added. Then the user will have quick access buttons/actions to run, test, deploy...

Tasks

Future work

• Know the difference between mcode and mcode-gpl. Asked about both in #377

Merged/done

• Fix GHDL 0.35-dev (<tarball>) [Dunoon edition]. ([build-system] do not require git in the Makefile #480)
• Fix distance to master. WONTFIX <v0.35, FIXED >=v0.25 (Releases not showing distance to master #483)
• GHDL GitHub organization (github.com/ghdl) (GHDL, a GitHub organization #500)
• Split tasks to
  • Deploy to Github releases
  • Push to dockerhub

PR #489

• Use multistage builds.
  • Merge dist/linux/docker/build-<platform>-<compiler> files to dist/linux/docker/build/<platform>
• Separate build and run images
  • Edit dist/linux/docker/build/<platform> and add dist/linux/docker/run/<platform>
• Create repositories at DockerHub:
  • ghdl/build
  • ghdl/run
  • ghdl/ghdl
  • ghdl/ext
  • ghdl/pkg
• Travis stages
  • Stage 0 Create build|run images
    • Run only if commit message contains [ci images]
    • Split build and deploy to dockerhub tasks
  • Stage 1 test [default]
    • Use matrix expansion
    • Split dist/linux/travis/buildtest.sh to dist/linux/travis/build.sh and dist/linux/travis/test.sh
    • Pull ghdl/build instead of buiding it from the Dockerfile
    • Execute build inside ghdl/build
    • Pull ghdl/run
    • Build ghdl/ghdl from ghdl/run and add the tarball generated in the build task
    • Execute test inside ghdl/ghdl
    • If successful:
      • Build ghdl/pkg
        • tarball generated in the build task
      • Push ghdl/ghdl and ghdl/pkg (if exists)
      • If tagged, push to GitHub releases
• Travis-DockerHub integration
  • Create dummy user
    • Create dummy email account
  • Add dummy user to GHDL organization
  • Encrypt user and password using travis CLI
    • Write shell script to automate: dist/linux/travis/travis-enc-dockerhub.sh
    • 1138-4EB/ghdl
    • ghdl/docker

PR #583

• Testable/reproducible issues (1138-4EB/issue-runner, Issue runner #579, Issue runner (attached zipfile) #584)
  • Add explanation to issue template suggesting users to test their code in a known environment: local, travis or play-with-docker

[Paebbels]

I don't like splitting up the documentation. For now, you should finish our documention rework started almost a year ago.

[eine]

[@Paebbels] I don't like splitting up the documentation.

I think it is getting too long...

@tgingold 2017-02-14]
The doc targets the end user. It is ok to have a chapter on building/installing ghdl (as well as a chapter on the internals), but if that gets too long, I think we'd better to have two doc.

---

For now, you should finish our documention rework started almost a year ago.

It is my bad, I thought it was partially merged. Indeed, it was almost ready, so I rebased and cleaned it up. See #482.

[eine]

I think it is ready to be reviewed now. At least Use cases for the actually available images, Let's have some fun! and Extending base images.

@flip111 @Nic30 the second demo is an example of how to get "parsing" info from GHDL. In the screen recording the VCD is used only, but you can see that many disp_tree*.txt files are created, along with adder.html and adder.xml. Note that you can mount the local dir in the container. This one liner will do it:

$(commnad -v winpty) docker run --rm -it -v "/$(pwd):/src" -w "//src" ghdl/ext:vunit bash -c " \
mkdir -pv elide \
&& curl -L https://github.com/1138-4EB/elide/archive/develop.tar.gz | tar xz -C elide --strip-components=1 \
&& ./elide/examples/full_adder/test.sh"

@Paebbels , please, let me know if any of the images fits your needs to run GHDL in CI environments.

[Paebbels]

@1138-4eb I'm currently out of sync in many of my projects. I try to get them in sync in the Christmas holidays.

[eine]

I moved some of the content from this issue to the new ghdl/docker repo. See: README.md, USE_CASES.md and ghdl/docker#2. Some other content was moved to 1138-4EB/ghdl-io (see 1138-4EB/ghdl-io#1).