Simplify installation procedure

[kboronski-ant]

The installation procedure is way too convoluted at the moment.

It involves:

• setting multiple environmental variables
• pullng a repo that shouldn't be even related to installation (f4pga-examples)
• manually creating and setting up conda environment
• downloading and extracting multiple archives
• installing local package through PIP

Then, to use the python wrappers one must enter the right conda environemnt (this will probably remain probably inevitable for quite a long time), set F4PGA_INSTALL_DIR and hope there are no missing dependecies (like click).

How IMO it should work:

Download an install bash script.
Run the script, eg.:

install --path /opt/f4pga --devices xc7a50t xc7a100t xc7a200t

The script updates conda environment and installs pip packages, including f4pga
The script downloads artifacts required for requested platforms.
The script exports settings.sh file with environmental variables to source in order to use f4pga.

Note that the management of artifacts could be done as a part of f4pga through modified (s)tdm.
See https://github.com/antmicro/f4pga/blob/f4pga-installer/f4pga/common/tdm.py, https://github.com/antmicro/f4pga/blob/f4pga-installer/f4pga/installer/__init__.py

This is related to #556

(note, this doesn't exactly work as intended because it uses environment.yml provided within packages which is invalid - currently we need to use one from f4pga-examples).

[umarcor]

• setting multiple environmental variables

This is related to using the deprecated wrappers vs using the f4pga build command. Supporting the deprecated flows is a requirement. Nonetheless, see the comment about f4pga-env below.

• pullng a repo that shouldn't be even unrelated to installation (f4pga-examples)

This is not a requirement. That's one solution for installing all the required tools. There are other approaches: https://github.com/hdl/packages. The procedure for installing f4pga itself, assuming that tools are available in the PATH, and using the new flows only, is just calling pip once.

• manually creating and setting up conda environment

See previous point.

• downloading and extracting multiple archives

#556.

• installing local package through PIP

If the environments from arch-defs or the updated guidelines from examples are used (see chipsalliance/f4pga-examples#306), installing f4pga manually/explicitly is not required. Some users might want to do it, though. It depends on how they want to install the tools/environment.

Then, to use the python wrappers one must enter the right conda environemnt (this will probably remain probably inevitable for quite a long time), set F4PGA_INSTALL_DIR and hope there are no missing dependecies (like click).

If any dependency is missing, that's a packaging issue that we should solve by adding it to https://github.com/chipsalliance/f4pga/blob/main/f4pga/requirements.txt. Setting F4PGA_INSTALL_DIR can be managed as a CLI option or as an option in the configuration file. Moreover, we might reconsider including f4pga-env in the arch-defs packages, which would allow us to get rid of all the envvars.

With regard to the conda environment itself, I refer to the comment above.

Download an install bash script. Run the script, eg.:

install --path /opt/f4pga --devices xc7a50t xc7a100t xc7a200t

This is similar to #561 (comment). So:

https://github.com/antmicro/f4pga/blob/umarcor/test/activate/.github/workflows/Pipeline.yml#L237-L252

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O conda_installer.sh
# Process the devices arg to decide the family?
export FPGA_FAM=xc7
bash conda_installer.sh -u -b -p $ARG_PATH/$FPGA_FAM/conda;
source "$ARG_PATH/$FPGA_FAM/conda/etc/profile.d/conda.sh";
# Which environment to use if not from arch-defs or f4pga-examples?
conda env create -f $FPGA_FAM/environment.yml
for PKG in install $ARG_DEVICES; do
  wget -qO- https://storage.googleapis.com/symbiflow-arch-defs/artifacts/prod/foss-fpga-tools/symbiflow-arch-defs/continuous/install/20220406-185509/symbiflow-arch-defs-${PKG}-3ef4188.tar.xz | tar -xJC $ARG_PATH/xc7/install
done
export PATH="$ARG_PATH/$FPGA_FAM/install/bin:$PATH"
conda activate xc7

Overall, I do understand the motivation. However, see #561 (comment).

(note, this doesn't exactly work as intended because it uses environment.yml provided within packages which is invalid - currently we need to use one from f4pga-examples).

I don't understand this comment. The tdm (arch-defs packages manager) should only take care of downloading and extracting tarballs in certain locations. It should not care about environments.

[kboronski-ant]

(note, this doesn't exactly work as intended because it uses environment.yml provided within packages which is invalid - currently we need to use one from f4pga-examples).

I don't understand this comment. The tdm (arch-defs packages manager) should only take care of downloading and extracting tarballs in certain locations. It should not care about environments.

The original, STDM does not even download packages and I don't think there's a clear guideline on the scope of what TDM should do (altough it could be understood intuitively that it should be used for downloading stuff). I do, however agree, that it shouldn't be responsible for installing anything. At no point I suggested that TDM should be used to install stuff. I did however attempt to install f4pga using the resources contained in artifacts (which were downloaded using a version of TDM that's included in my fork, but that's besides the point). The install artifact contains environment.yml file which I mistakenly tried to use to update conda environment. This however seems to be a leftover from arch-defs and is irrelevant for the end-user. I don't think it can be used for anything. The comment was meant to point out that the script I wrote and linked contains this mistake: using environment.yml extracted from artifacts and shouldn't be used as a reference on how to install F4PGA.

This is not a requirement. That's one solution for installing all the required tools.

Correct me if I'm wrong, but that's the only solution mentioned in the docs that's not targeted towards people who want to develop arch-defs. The fact that I could install all the tools manually (assuming I'm familiar with the entire tech stack used by f4pga, which might not be the case for newcomers), use other repos, etc. doesn't make the whole process more straight-forward.

setting multiple environmental variables

This is related to using the deprecated wrappers vs using the f4pga build command. Supporting the deprecated flows is a requirement. Nonetheless, see the comment about f4pga-env below.

I'm pretty sure that this can be solved by generating settings.sh (similar to how Vivado handles stuff like that - source settings.sh before running Vivado and you are good to go). Those exports aren't really needed during the installation, are they?

Moreover, we might reconsider including f4pga-env in the arch-defs packages, which would allow us to get rid of all the envvars.

Not sure where it's used, but it looks useful for improving installation procedure. On the other hand, how hard would it be to make these paths follow a uniform scheme?

If the environments from arch-defs or the updated guidelines from examples are used (see chipsalliance/f4pga-examples#306), installing f4pga manually/explicitly is not required. Some users might want to do it, though. It depends on how they want to install the tools/environment.

This PR is in a draft state and seems to be related to f4pga-examples. One the points I'm trying to make is that the installation process should be be made easy without relying on the the examples or arch-defs repo.

This is similar to #561 (comment). So:
https://github.com/antmicro/f4pga/blob/umarcor/test/activate/.github/workflows/Pipeline.yml#L237-L252

Overall, I do understand the motivation. However, see #561 (comment).

Tuttest shouldn't be an issue if you have a working stand-alone script that you can reference in a code-block within a tutorial.

---

I also realize that managing conda environments might be tricky and conda will complain about not being initialized for a given shell when it's not explicitly ran by the end-user. I don't know how this could be handled correctly at the moment. My suggestion, however would be to at least separate conda installation procedure from the rest (creating environment, setting environmental variables, etc). in getting started docs. Maybe just list it as a prerequisite for using F4PGA. It's confusing to read the instructions when you already have conda installed on your system.

[umarcor]

The original, STDM does not even download packages and I don't think there's a clear guideline on the scope of what TDM should do (altough it could be understood intuitively that it should be used for downloading stuff).

Fair enough. Yet, note the description of SymbiFlow/symbiflow-tools-data-manager: "Python based package manager to get packages and artifacts of symbiflow projects". See also olofk/pdk-config.py.

I did however attempt to install f4pga using the resources contained in artifacts (which were downloaded using a version of TDM that's included in my fork, but that's besides the point). The install artifact contains environment.yml file which I mistakenly tried to use to update conda environment. This however seems to be a leftover from arch-defs and is irrelevant for the end-user. I don't think it can be used for anything. The comment was meant to point out that the script I wrote and linked contains this mistake: using environment.yml extracted from artifacts and shouldn't be used as a reference on how to install F4PGA.

I believe we need to agree on clarifying the terminology, because this is going to be confusing for us and for users. When you say "attempt to install f4pga" do you mean the same as with "how to install F4PGA"? Do you use the lowercase to refer to the Python package and the uppercase to the whole environment (collection of tools)? We might use pyfpga to refer to the Python package.

Overall, the f4pga Python package does not need to be installed inside the conda environment. It is recommended to always use pip inside a Python virtual environment, and Conda is a particular type of such environment. Therefore, it is sensible to install the f4pga Python package inside Conda. However, it is not a requirement. Ideally, the f4pga CLI would be able to interact with tools installed using various procedures (see issues 15, 16 an 17 from SymbiFlow/make-env). E.g. f4pga build --env=path/to/conda/installation or f4pga build --env=path/to/container/list.yml (see https://github.com/olofk/edalize/blob/master/scripts/el_docker#L13-L75) or f4pga build --env=system (to use the tools available in the PATH). Hence, users might want to install the f4pga Python package once only (maybe using option --user to have it located in the home directory, without admin permissions).

Anyway, your comments about the environment.yml located in the arch-defs packages are revealing. I was not aware of that. It's not exactly a leftover and it is relevant indeed. I was aware that there were scripts used in two "abstraction layers" of the arch-defs packages (the higher level shell scripts which we moved here, and the lower level python utils). However, I thought that all of them were plain scripts and did not have further dependencies. As such, I was considering the current arch-defs packages to be installable as plain tarballs. You made me realize that is not the case. See f4pga/f4pga-arch-defs#2696.

Correct me if I'm wrong, but that's the only solution mentioned in the docs that's not targeted towards people who want to develop arch-defs.

That depends on the scope. If we focus on the F4PGA repositories, that is correct (see chipsalliance/f4pga-examples#227). However, there are other people in the broader community providing alternatives. Edalize is one of them (see f4pga/ideas#66, https://github.com/f4pga/ideas/blob/main/gsoc-2022-ideas.md#f4pga-toolchain-integration-in-mainline-edalize and https://github.com/f4pga/ideas/blob/main/gsoc-2022-ideas.md#generalization-of-wrapper-scripts-for-installed-f4pga-toolchain-and-making-them-OS-agnostic), which supports using the containers built in hdl/containers. See comments below about the "Getting started" section.

The fact that I could install all the tools manually (assuming I'm familiar with the entire tech stack used by f4pga, which might not be the case for newcomers), use other repos, etc. doesn't make the whole process more straight-forward.

That is correct. However, the whole idea is that you don't need to do it manually; someone else does it for you. Typically, that role is the one of "package" and "distribution" maintainers. Conda is one package management and distribution solution, but there are many other. Most system package managers do handle "groups" of utilities. For instance, on MSYS2, group mingw-w64-x86_64-eda allows installing ~20 tools at once. On Fedora, there is coprs/rezso/HDL (see hdl/containers#41).
hdl/containers is another packaging and distribution solution. As quoted in chipsalliance/f4pga-examples#227:

The whole point of the containers is a little bit that you do not care whats inside as long as it works. Its a glorified "wget a big rootfs + chroot + forget" in many cases.

Which, in the context of F4PGA, is pretty similar to how most users expect to use Conda.
As documented in https://hdl.github.io/containers/ug/AllInOne.html#f4pga, by using containers, users don't need to care about environment variables. Find syntax variations in https://github.com/hdl/packages/blob/main/.github/workflows/containers-conda-f4pga.yml.

NOTE: Conda containers will break when chipsalliance/f4pga-examples#306 is merged, and I will need to fix the recipe.

NOTE: As discussed in olofk/edalize#299, because of the size of arch-defs packages, @olofk was considering to use the container with the "install" package only, and then "dynamically" fetch the arch-defs packages for the desired architectures, keeping some kind of cache. I believe that's the context of olofk/pdk-config.py.

I'm pretty sure that this can be solved by generating settings.sh (similar to how Vivado handles stuff like that - source settings.sh before running Vivado and you are good to go).

If you mean adding it to f4pga-examples, I refer to #561 (comment). I am not aware of the criteria that lead to writing the commands one-by-one in the f4pga-examples documentation and then running them through tuttest. I can only guess... the purpose was to force the users to write/copy the commands and take care while doing so, which is a typical strategy for learning. I agree it does make sense to gather all the content of https://f4pga-examples.readthedocs.io/en/latest/getting.html in a single shell script which users can fetch and execute with a few options. However, that'd be less friendly to the very less experienced users (comments in shell VS a website). On the other hand, I would not recommend duplicating the guidelines (having the step-by-step tuttest approach in parallel to an all-in-one shell script); I'd neither oppose as long as both solutions are tested in CI.

If you mean adding it to this repo, see comments below about https://github.com/chipsalliance/f4pga/tree/main/.github/scripts.

Those exports aren't really needed during the installation, are they?

That is correct. The only exports needed before and during the installation are INSTALL_DIR and F4PGA_FAM. However, we should change INSTALL_DIR to F4PGA_INSTALL_DIR as soon as possible, and let the same envvar be used during the installation and by the f4pga build command. Furthermore, we might enhance https://github.com/chipsalliance/f4pga/blob/main/f4pga/wrappers/sh/__init__.py to get F4PGA_ENV_BIN and F4PGA_ENV_SHARE from F4PGA_INSTALL_DIR, unless explicitly defined. That would simplify the guidelines for users to only defining two environment variables for both installation and runtime.

Not sure where it's used, but it looks useful for improving installation procedure. On the other hand, how hard would it be to make these paths follow a uniform scheme?

It is not used at the moment. See 80748d7 and fbf7a4d. See also log of the Apr 19 meeting.

Nevertheless, f4pga-env would only remove the need of F4PGA_INSTALL_DIR, provided that f4pga-env is made available in the PATH. The proposal was to make f4pga-env part of the arch-defs packages. So, independently of how the environment is setup (Conda, containers, other...), the user would need to first "activate" it. When using Conda, the INSTALL_DIR is required in order to source conda.sh before activating it, so we could not get completely rid of it. Conversely, in containers it would be preset.

Overall, for now I think we'd better go with enhancing f4pga/wrappers/sh/__init__.py as commented above. f4pga-env might be more useful for other use cases which don't have full control over the environment (see f4pga/f4pga-arch-defs#2373). So, we might add the functionality into f4pga env, while relying on a single F4PGA_INSTALL_DIR.

This PR is in a draft state and seems to be related to f4pga-examples. One the points I'm trying to make is that the installation process should be be made easy without relying on the the examples or arch-defs repo.

The draft state was due to hardware-ci needing a fix. I wanted to point out the addition of f4pga to the requirements file (https://github.com/chipsalliance/f4pga-examples/pull/306/files#diff-b0fc49199a26dad2e2805f182532239409c487a34c30124bb6aeb39c43fefbda), as a reply to "installing local package through PIP".

Anyway, I agree it would be interesting to have guidelines for intermediate users about how to get the flows running faster. Maybe the documentation of this repo is the place for such content? We can have a reduced set of tests/examples with a simplified environment and a couple of shell scripts. In the end, f4pga-examples is focused on testing many examples on many platforms, and arch-defs is for developing the packages.

I also realize that managing conda environments might be tricky and conda will complain about not being initialized for a given shell when it's not explicitly ran by the end-user. I don't know how this could be handled correctly at the moment. My suggestion, however would be to at least separate conda installation procedure from the rest (creating environment, setting environmental variables, etc). in getting started docs.

This is something I have in my notes for enhancing https://f4pga.readthedocs.io/en/latest/getting-started.html:

• Users (see F4PGA-examples)
• Developers (see F4PGA-arch-defs)
• Toolchain installation
  • Recommended: Conda
  • Other:
    • hdl/packages
    • hdl/containers

Note that https://github.com/chipsalliance/f4pga/blob/main/.github/scripts/activate.sh is rather close to the settings.sh you mentioned. See also https://github.com/chipsalliance/f4pga/blob/main/.github/scripts/prepare_environment.sh (L77-L106 are to be removed when chipsalliance/f4pga-examples#306 is merged).

[umarcor]

Closing as completed in #588. Pending issues, such as the pdk-config/stdm or using containers through an Action, will be dealt with in follow-up issues/PRs.