Support specifying which languages to install/skip

[edmorley]

Hi!

We have an app that is comprised of a webpack-built SPA and a django-rest-framework powered backend - with both parts existing in the same repository (monorepos have many advantages):
https://github.com/mozilla/treeherder

We would only want the SPA to be built by Netlify (so only want the nodejs env setup + yarn install steps to be run), however run-build-functions.sh also finds runtime.txt and requirements.txt, resulting in unwanted (and failing) Python setup tasks being run:

7:13:05 PM: Installing dependencies
7:13:05 PM: /usr/local/bin/run-build-functions.sh: line 150: /opt/buildhome/pythonpython-2.7.14/bin/activate: No such file or directory
7:13:05 PM: Error setting python version from runtime.txt
...
7:13:08 PM: Installing pip dependencies
...
7:13:12 PM: Collecting mysqlclient==1.3.12 (from -r requirements/common.txt (line 74))
7:13:12 PM:   Downloading mysqlclient-1.3.12.tar.gz (89kB)
7:13:12 PM:     Complete output from command python setup.py egg_info:
...
7:13:12 PM:     EnvironmentError: mysql_config not found
7:13:12 PM:     ----------------------------------------
7:13:13 PM: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-PcfhfO/mysqlclient/
7:13:13 PM: Error installing pip dependencies
7:13:13 PM: Error running command: Build script returned non-zero exit code: 1
7:13:13 PM: Failing build: Failed to build site

The "install specific Python version" step is giving a warning, since our runtime.txt is for use by heroku-buildpack-python and so uses their python-N.N.N format rather than that used by Netlify (ie has the additional python- prefix).

The pip install step is failing since the mysqlclient package requires that the libmysqlclient-dev package be installed in the OS.

However even if these two steps succeeded, they would only be unnecessarily increasing our build times, since our SPA has no need for the Python environment.

As such we would like a way to prevent them from being run.

Perhaps a new language option, like:

# netlify.toml
[build]
  language = "node"
  publish = "dist"
  command = "yarn build"

...which would skip all steps other than those relating to node/js dependencies.

Or else a LANGUAGES environment variable would be fine too.

Many thanks! :-)

[brycekahle]

@edmorley I agree that specifying the desired language would be a good optimization to make. It would allow us to skip other language version switches/installs and caching of language-specific dependencies. It would definitely need to be an opt-in behavior, because we strive to make our build-image mostly automatic out-of-the-box.

Since most of that behavior is contained in the run-build-functions.sh script, it would likely end up being an environment variable populated by our netlify.toml parser. LANGUAGES is very close to LANGUAGE (a locale env var). It may be prudent to prefix, say NF_LANGUAGES.

I'll try and solicit the opinions of some of my colleagues and see where this falls on our roadmap.

[edmorley]

Thank you for the reply :-)

I agree this would need to be opt in, and that it would be a nice optimisation for all that do so.

Though just want to clarify that in our case it was less about the performance, and more that as-is Netlify is unusable on our monorepos due to the unavoidable build failures (the only workaround being to rearrange our repository layouts, and use base = "some-subdirectory", but that would then break other tooling/bots that expect files to be in the repo root).

I was thinking something like this might work (and also be a nice cleanup of the rather large run-build-functions.sh):

# run-build-functions.sh
DIR="$(dirname "${BASH_SOURCE[0]}")"

# Default to all languages, for automatic detection out of the box
: "${NETLIFY_LANGUAGES:='node python ruby ...'}"

# non-language specific prep steps here (such as restoring home/cwd cache)

for lang in ${NETLIFY_LANGUAGES}; do
    if [[ -f "${DIR}/languages/${lang}.sh" ]]; then
        . "${DIR}/languages/${lang}.sh"
    else
        echo "Unrecognised language '${lang}' in NETLIFY_LANGUAGES, see <docs URL>"
        exit 1
    fi
done

# non-language specific teardown steps here (such as saving home/cwd cache)

...and then each language's script file contains the specific cache prep / binary install / dependency install / cache save steps.

Or if separate files weren't desired, then inline functions and something like:

for lang in ${NETLIFY_LANGUAGES}; do
    if command -v "setup_${lang}" >/dev/null 2>&1; then
        "setup_${lang}"
    else
        echo "Unrecognised language '${lang}' in NETLIFY_LANGUAGES, see <docs URL>"
        exit 1
    fi
done

[brycekahle]

Though just want to clarify that in our case it was less about the performance, and more that as-is Netlify is unusable on our monorepos due to the unavoidable build failures

I did see that. Unfortunately, I don't see a quick way to fix that besides what you already did.

and then each language's script file contains the specific cache prep / binary install / dependency install / cache save steps

The one trick here is that we currently support multiple languages at the same time, so the function names couldn't overlap.

[sover02]

Just for the record, I ran into this today and it's preventing me from using the service. :(

Our frontend is built with node and can be served as a static html/css. I'd like to build and host this with netlify.
Our backend is built with python and hosted on heroku, which uses the python-3.7.2 format, while netlify uses python-3.7.

Both services are hard-fast fixed on their format, so we cannot have a runtime.txt that works for both. Having the ability to skip the runtime.txt) step would allow us to build and serve the frontend, but builds fail with netlify when it cannot source the virtualenv when concatenating the contents of runtime.txt, and we can't disable the functionality even if we don't use it

Being able to use and override the info with an environment variable (like netlify does with node and ruby) would also solve this for us.

Old post with the same use case:
https://community.netlify.com/t/runtime-txt-format-incompatible-with-heroku/1926

[edmorley]

(Sorry previously commented on the wrong tab! I've moved that comment to: heroku/heroku-buildpack-python#913 (comment))

Our backend is built with python and hosted on heroku, which uses the python-3.7.2 format, while netlify uses python-3.7.

@sover02 Hi! Since I filed this issue I've now become the Python owner at Heroku, and as such am in a position to try and improve this from Heroku's side. We want to add "flexible versions" support to the Heroku Python buildpack (see heroku/heroku-buildpack-python/issues/913), which will bring the version specifications for Heroku and Netlify closer in line.

However I did notice that in addition to the X.Y vs X.Y.Z version format differences, Netlify requires that there be no python- prefix. We may end up supporting the prefix-less form too, however for completeness perhaps the Netlify build script could also be tweaked to remove the prefix if it exists too?

[sover02]

@edmorley, that's fantastic! I'm excited to possibly make use of both services :)

For what it's worth, I think a simple solution is sight on the netlify side - we could just allow runtime.txt to be ignored. :)

[cmdlinebeep]

Did this ever get anywhere? I'm in the same boat where my repo has components on Netlify and others on Heroku.

Can one or both please be fixed to be a little more tolerant?

"Be liberal in what you accept" and all that...

Thanks! :-)

[cmdlinebeep]

@edmorley Hi Ed, are you still in a position at Heroku to fix this? I know it's been over three years since you opened this, but it's still causing issues for folks today (like me!)

In lieu of some complicated change to the way we specify our runtime versions, it seems like it should be straightforward to make the parser support either:

X.Y.Z
python-X.Y   or
python-X.Y.Z

patterns in the runtime.txt without breaking any existing Heroku runtime.txt files.

Here is the regexp, can we please get it in? 🙏

/(\d+\.\d+(\.\d+)?)/

I tested this at regexr.com. Thanks!

[edmorley]

@cmdlinebeep Hi! I replied over in heroku/heroku-buildpack-python#1243 (comment) but to summarise here...

Given the filename is called runtime.txt (ie: it is not language specific), it feels wrong to put version numbers in there without the name of the language runtime.

I think a better change would be to update this line:

build-image/run-build-functions.sh

Line 225 in fa439ad

PYTHON_VERSION=$(cat runtime.txt)

With:

    PYTHON_VERSION=$(cat runtime.txt)
    # Remove any `python-` prefix from the version string.
    PYTHON_VERSION="${PYTHON_VERSION#python-}"

With that change, Netlify will work with any of the thousands of repos that have the existing style runtime.txt (runtime.txt is a concept that only started because of the Heroku Python buildpack, so if it's going to be re-used, it really should support the existing contents IMO).

[edmorley]

Also, I think it would be worth splitting the runtime.txt parsing issue into its own GitHub issue in this repo, as this issue as filed is about something slightly different (and harder to attain).