FireSim CI (#409)

* sketch of firesim ci

* cleanup + first test (add click to continue guards)

* init-manager immediately | have remote directory not be based on job

* fix sourceme | fix heredoc

* update tests to what Nathan recommends | make work dir first

* have approvals be seperate jobs | add cd to home after run

* retrieve cache for run workloads | get defaults in hash gen script

* update hash creation

* sketch of managing collateral

* adjust rsync folders

* fix permissions and delete extra files

* use firesim_configs to specify firemarshal workload

* make rsync not query | point to firesim firemarshal

* get toolchain for marshall

* build firemarshal jobs serially | fix bug in buildafi pointers

* clean dirs on failure and at beginning of init

* rm before clone

* copy over boom files correctly

* update timeout for afi

* update defaults

* spawn new manager instance | update runfarm tags

* try to use curls to spawn afis/workloads

* testing: use short afi build

* add the api url to do curls

* proper url path | debug screen list

* fix detach script func

* update comments

* update run detach script

* add remove screen list at end

* run both workloads | pass in arguments correctly | update runtimes to use right afi

* reorganize files | more variables | cleaner comments

* testing: skip afi build - use previously made afi

* fix config

* use key value store method to get afinames | fix finish job bug | more specific job names

* fix up logging on finish/failures

* deal with race conditions in firesim make

* escape flock and put lock in writable location

* rebuild afi | bump to dev chipyard | setup normal schedule | swap to boom afi

* cleanup firesim configs | update docs more

* test: generalized launch workloads based on afi | re-test with rocket

* test: use boom afi

* revert to weekly schedule | updated config names

* add coremark ci job

* run ci now with jerry fix

* test: use older afi

* first pass on adding spec

* add timeout parameter

* skip over certain directories

* [ci] use spot instances for runs | auto-terminate on finish

* try to launch manager instance using cli [no spec on manager]

* bump CHIPYARD.hash

* fix config.yml syntax

* remove aws context

* pass in credentials to firesim managerinit

* fix misc errors

* bump

* build spec on manager instance

* update

* add expect script on spec install

* correctly setup spec | cache instance file in common (l)unix location | update SPEC17 intrate runtime

* disable spec17 runs

* add rsync to all aws jobs | use rocket for now

* update firesim rocket config

* migrate to CircleCi v2 API

* first pass on using command (functions)

* fix command syntax | more abstraction | misc. fixes

* update comments

* bump CHIPYARD.hash

* fix finish workload curl

* fix finish workload | swap back boom | run all workloads

* add override for run-workload | put back original ci

* temporarily disable spec

* fix csmith bug | parameterize config more

* abstract out executor

* simplify cache bumps

* [ci] remove smallboomandrocket tests

* [ci] bump ci docker image

* [ci] fix firesim names | add more comments to scripts

* [ci] use verilator on build machines

* [ci] use on demand instances | cleanup

* update firesim manager setup instructions

* debug: remove schedule for firesim testing

* bump aws orb and chipyard image

* debug: stop running default tests

* run upgrade before rsync download

* workaround: install git on manager instance

* update the naming of the afi longname

* update before rsync install | patch firesim sourceme

* update comments

* source chipyard env before sourceme

* bump chipyard and use newer firesim version

* bump init manager

* change path to expect

* bump manager init

* checkout right firesim version in init steps

* bump to newer cy | remove separate fsim update

* bump firesim | remove need to manually install expect

* bump firesim | remove unneeded lines

* re-log immediately after machine-launch completes

* bump chipyard

* bump chipyard to test firesim fix

* update timeout and fix stop-instance-check

* update timeout

* bump firesim | delete extra 5GB volume

* bump vol delete

* set xterm terminal

* try to set term from config

* print environment and switch to c5.4

* bump

* use @aou's bumped firesim | misc cleanup

* add afi bypass

* bump wait time

* silent rm fail on debug

* move results-dir to place where copy wont fail | bump cy

* re-enable firesim-ci on a schedule | small updates to README

* misc cleanup | test new way to detach 5GB volume

* re-add the cron schedule

* delete unneeded code

* remove approval button on normal ci

* clarify what circleci env. vars. mean

* attempt to speed things up a bit

* reduce amount of make processes for build toolchains

.circleci/README.md

@@ -1,15 +1,54 @@
+CI Notes
+----------------
+
 These are the scripts that Circle CI uses to run the tests during a PR.
 
-Note: This uses the most up to date version of ucb-bar/project-template (**boom-ci**) to run the tests.
-Thus this requires the **boom-ci** branch to have all the changes needed for boom-ci to work.
+Note: This uses the Chipyard `ucb-bar/chipyard:dev` branch to run CI (or other similar commits).
 
-WARNING: IF **boom-ci** BRANCH IN ucb-bar/project-template IS GONE. CONTACT THE BOOM DEVELOPERS!!!!
+Note: CircleCI env. var. refers to adding a environment variable by navigating to the "Project Settings" in the web UI and adding
+an environment variable in the "Environment Variables" tab.
 
-Note: This uses $SERVER and $CI_DIR which is given in the CircleCI env var setup to specify a server to build on.
+Note: This uses `$SERVER` and `$CI_DIR` which is given in the CircleCI env. var. setup to specify a server to build on.
 To change these variables you must change the project settings of CircleCI.
 
 Note: You also need to add the private key of the build server to CircleCI and match the fingerprint it gives in the config.yml
 
 Things to look into:
 --------------------
 * How to get more coverage of pipeline using something like csmith and/or riscv-torture.
+
+FireSim CI Notes
+----------------
+
+Currently only supports 1 AFI to build and test so that the manager stop procedure works.
+
+Stop procedure:
+- When an AFI fails, it will go ahead and stop the instance then stop the CI.
+- When all workloads for an AFI have finished, the manager will stop.
+
+Terminate procedure:
+- The manager instance is never terminated (for debugging purposes). Make sure to terminate if unneeded.
+
+Requirements:
+- Add to CircleCI env. var. `$CI_AWS_DIR` point to the "~" of manager instance
+- Add SSH key ("firesim.pem") to the CircleCI SSH keys and use the key in the `config.yml` jobs
+- Add `$API_TOKEN` to the env. vars in CircleCI - make sure it is the user API token (found in user settings, not in project settings)
+- Fill out the following CircleCI env. var.s to access AWS:
+    - `$AWS_ACCESS_KEY_ID`
+    - `$AWS_SECRET_ACCESS_KEY`
+    - `$AWS_DEFAULT_REGION`
+- Add your `firesim.pem` into the `$FIRESIM_PEM` CircleCI env. var.
+    - Note: This is a single line .pem where all \n's are ,'s
+
+If the workloads fail to run AND the manager instance stops, you can still run tests again:
+    - Re-boot the instance (note: the IP address has changed)
+    - Add the CircleCI env. var. `$AWS_IP_ADDR_OVERRIDE` with the new IP address
+    - Re-run the tests
+    - REMEMBER TO REMOVE THE ENVIRONMENT VARIABLE ONCE DONE! OTHERWISE THE CI WILL BREAK ON THE WORKLOAD STEPS
+
+You can also re-use a built AFI:
+    - Add the CircleCI env. var. `${CONFIG_KEY}_OVERRIDE` with the agfi id (i.e agfi-...) where `${CONFIG_KEY}` is the AFI-type to override
+    - Re-run the tests
+    - Note: The 1st infrasetup will take an extremely long time since it is building the RTL/driver for the 1st time
+    - REMEMBER TO REMOVE THE ENVIRONMENT VARIABLE ONCE DONE! OTHERWISE THE CI WILL USE A STALE AFI
+

.circleci/build-run-csmith-tests.sh

@@ -1,6 +1,13 @@
 #!/bin/bash
 
+#-------------------------------------------------------------
 # test the verilator simulation using csmith random testing
+#
+# run location: circle ci docker image
+# usage:
+#   $1 - config string
+#   $2 - name of the toolchain to build
+#-------------------------------------------------------------
 
 # turn echo on and error on earliest command
 set -ex

.circleci/build-toolchains.sh

@@ -1,6 +1,12 @@
 #!/bin/bash
 
+#-------------------------------------------------------------
 # create the riscv tools binaries from ucb-bar/chipyard with rocket-chip hash given by riscv-boom
+#
+# run location: circle ci docker image
+# usage:
+#   $1 - name of the toolchain to build
+#-------------------------------------------------------------
 
 # turn echo on and error on earliest command
 set -ex
@@ -21,6 +27,6 @@ if [ ! -d "$HOME/$1-install" ]; then
 
     cd $HOME
 
-    # init all submodules including the tools
-    CHIPYARD_DIR="$LOCAL_CHIPYARD_DIR" NPROC=2 $LOCAL_CHIPYARD_DIR/scripts/build-toolchains.sh $1
+    # init all submodules including the tools (doesn't use CI_MAKE_PROC due to mem. constraints)
+    CHIPYARD_DIR="$LOCAL_CHIPYARD_DIR" NPROC=8 $LOCAL_CHIPYARD_DIR/scripts/build-toolchains.sh $1
 fi

.circleci/clean-old-files.sh

@@ -1,29 +1,49 @@
 #!/bin/bash
 
+#-------------------------------------------------------------
 # clean directories that are older than 30 days
-# argument is used as the directory to look in
+#
+# run location: build server (located at UCB)
+# usage:
+#   $1 - the directory to search in
+#-------------------------------------------------------------
 
+#-------------------------------------------------------------
+# get age of the folder in days
+#
+# usage:
+#   $1 - the folder to get the age of
+#-------------------------------------------------------------
 age () {
-	local AGE_SEC
-	local CUR_SEC
-	local DIFF_SEC
-	local SEC_PER_DAY
+    local AGE_SEC
+    local CUR_SEC
+    local DIFF_SEC
+    local SEC_PER_DAY
 
-	SEC_PER_DAY=86400
+    SEC_PER_DAY=86400
 
-	CUR_SEC=$(date +%s)
-	AGE_SEC=$(stat -c %Y -- "$1")
-	DIFF_SEC=$(expr $CUR_SEC - $AGE_SEC)
+    CUR_SEC=$(date +%s)
+    AGE_SEC=$(stat -c %Y -- "$1")
+    DIFF_SEC=$(expr $CUR_SEC - $AGE_SEC)
 
-	echo $(expr $DIFF_SEC / $SEC_PER_DAY)
+    echo $(expr $DIFF_SEC / $SEC_PER_DAY)
 }
 
 for d in $1/*/ ; do
-	DIR_AGE="$(age $d)"
-	if [ $DIR_AGE -ge 30 ]; then
-		echo "Deleting $d since is it $DIR_AGE old"
-		rm -rf $d
-	else
-	    echo "Keep $d since it is $DIR_AGE old"
-	fi
+    DIR_AGE="$(age $d)"
+    if [ $DIR_AGE -ge 30 ]; then
+        if [[ $d == *"bin"* ]]; then
+            echo "$d is getting skipped"
+            continue
+        fi
+        if [[ $d == *"config"* ]]; then
+            echo "$d is getting skipped"
+            continue
+        fi
+
+        echo "Deleting $d since is it $DIR_AGE old"
+        rm -rf $d
+    else
+        echo "Keep $d since it is $DIR_AGE old"
+    fi
 done