netlify.toml written at "postinstall" not detected by @netlify/build

[juanpprieto]

Describe the bug
Hi team. not sure if this is the correct place, but I hope I can find some pointers. Link to issue on the support forum.

As the title says, I am trying to inject a netlify.toml which includes custom plugins before netlify/build, but it doesn't seem to be detected (i.e looks like its too late for it)

I'm doing this because I want to stop endusers from being able to modify the TOML and because I don't want the file to be tracked in git so that the endusers dont have to know the inner workings of our product.

Is there any way to write/inject a netlify.toml before the netlify build / config sequence kicks in?

Configuration

"postinstall" script - writes our custom netlify.toml to the root folder i.e /opt/build/repo

"postinstall": "node --eval \"require('@my-org/netlify-toml')\""

Deploy logs

7:39:51 AM: [5/5] Building fresh packages...
7:39:57 AM: $ node --eval "require('@myorg/netlify-toml')"
7:39:59 AM: Created default netlify.toml at /opt/build/repo <------ ✅
7:39:59 AM: Done in 61.65s.
7:39:59 AM: NPM modules installed using Yarn
7:39:59 AM: Started restoring cached go cache
7:39:59 AM: Finished restoring cached go cache
7:39:59 AM: go version go1.14.4 linux/amd64
7:39:59 AM: go version go1.14.4 linux/amd64
7:39:59 AM: Installing missing commands
7:39:59 AM: Verify run directory
7:40:00 AM: ​
7:40:00 AM: ────────────────────────────────────────────────────────
7:40:00 AM:   Netlify Build                                                 
7:40:00 AM: ────────────────────────────────────────────────────────
7:40:00 AM: ​
7:40:00 AM: ❯ Version
7:40:00 AM:   @netlify/build 17.9.2
7:40:00 AM: ​
7:40:00 AM: ❯ Flags
7:40:00 AM:   baseRelDir: true
7:40:00 AM:   deployId: 61128f70de851d43819b2cb8
7:40:00 AM: ​
7:40:00 AM: ❯ Current directory
7:40:00 AM:   /opt/build/repo
7:40:00 AM: ​
7:40:00 AM: ❯ Config file <--- 🚫 Not detecting the netlify.toml we just wrote
7:40:00 AM:   No config file was defined: using default values.
7:40:00 AM: ​
7:40:00 AM: ❯ Context
7:40:00 AM:   production
7:40:00 AM: ​
7:40:00 AM: ❯ Loading plugins <---- 🚫 Not detecting our netlify.toml build plugins
7:40:00 AM:    - @netlify/plugin-nextjs@3.8.0 from Netlify app and package.

*Deploy Log
Full failing deploy logs @ deploy id 611316d8bda9b800087268ff

[juanpprieto]

As a test, I just attempted to copy a netlify.toml from a local subfolder on preinstall without any luck. (Still not being detected

2:05:24 AM: Finished restoring cached yarn cache
2:05:25 AM: No yarn workspaces detected
2:05:25 AM: Started restoring cached node modules
2:05:25 AM: Finished restoring cached node modules
2:05:25 AM: Installing NPM modules using Yarn version 1.22.10
2:05:25 AM: yarn install v1.22.10
2:05:25 AM: info No lockfile found.
2:05:25 AM: $ cp  ./netlify-toml/netlify.toml ./netlify.toml <---- ✅
2:05:25 AM: [1/5] Validating package.json...
2:05:25 AM: [2/5] Resolving packages...
2:05:29 AM: warning @backpackjs/api-backpack > @backpackjs/clients > datocms-client > request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
2:05:30 AM: warning @backpackjs/api-backpack > @backpackjs/clients > datocms-client > request > har-validator@5.1.5: this library is no longer supported
2:05:30 AM: warning @backpackjs/api-backpack > @backpackjs/clients > datocms-client > request > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older vers

later on the logs

2:06:16 AM: ❯ Current directory
2:06:16 AM:   /opt/build/repo
2:06:16 AM: ​
2:06:16 AM: ❯ Config file
2:06:16 AM:   No config file was defined: using default values. <---- 😢
2:06:16 AM: ​
2:06:16 AM: ❯ Context
2:06:16 AM:   production
2:06:16 AM: ​
2:06:16 AM: ❯ Loading plugins
2:06:16 AM:    - @netlify/plugin-nextjs@3.8.0 from Netlify app and package.json
2:06:17 AM: 

Again the goal is to use a netlify.toml without it being tracked on the repo.

Thanks for the help!
JP

[ehmicky]

Hi @juanpprieto,

Unfortunately, this is not currently possible. netlify.toml itself is read at the very beginning of the build, much before @netlify/build starts and before npm install is performed. This is because the process of performing npm install and @netlify/build depends on the contents of netlify.toml. For example, if there is a base directory in netlify.toml, this will change the current directory used by npm install. As a consequence, creating or updating that file directly is not supported at the moment.

That being said, we've just been adding support for modifying most Netlify configuration properties using Build plugins. This comment explains the details of it. It is also possible to read the Netlify configuration inside Build plugins. I am wondering whether those features might help you work around this limitation?

[juanpprieto]

Hi @ehmicky thanks for the quick response!

That's sort of what I thought, and does make a bit of sense - /base etc

Saying that, base can be easily configured via the UI. I'm more concerned about abstracting/hiding away non UI friendly settings, so that the end-user has no control/visibility on changing :

• our multiple [[plugins]] and their configs
• our [functions] configs

i.e The things you can't configure via the UI.

Interesting you mention this new feature you been working on.

I randomly came across it yesterday while scavenging for clues and ideas. From what I could gather on my brief read, I thought it could potentially help me overwrite basic build and functions settings, but I didn't see an option for managing plugins. Is this the case?

The best case scenario I came up (with my understanding) of the feature would be:

To track a more basic TOML which includes just a custom "orchestrating plugin". Then on it's onInit hook it would set/overwrite all the important build settings and more importantly add/remove the other custom plugins required for our builds.

This last part seems like it just wouldn't work? i.e registering plugins once the lifecycle is init seemed def unprovable.

Thanks in advance for your help!

P.S - I guess another option is to merge all our custom plugins into one and then just have a toml with this one plugin that would then run all our plugin features and control the settings of the netlifyConfig

[ehmicky]

I didn't see an option for managing plugins. Is this the case?

Yes, you're correct, plugins is one of the few properties not supported yet. Modifying plugins inside plugins raises many quite tricky dependency questions. Also, plugins are currently all spawned in parallel at the beginning of the build for both performance and logical reasons. So we've currently been leaving out modifying plugins inside plugins, although we might want to find a way to implement this eventually.

Saying that, base can be easily configured via the UI. I'm more concerned about abstracting/hiding away non UI friendly settings

To better understand your situation, you do want to allow users to override any build settings exposed in the UI, but not the ones available only in netlify.toml, is that correct? Note: Build plugins can also be added/removed from the UI (changing inputs is not possible yet though).

P.S - I guess another option is to merge all our custom plugins into one and then just have a toml with this one plugin that would then run all our plugin features and control the settings of the netlifyConfig

Would this work for your use case? Are there downsides to it?

[juanpprieto]

So, the goal is to not have a TOML tracked in the repo.

Why ?
The repo in question will function as a "starter" template. This template will be the starting point for all our clients and internal projects and we want to make sure all projects are "locked" to what we decide is a correct TOML config. This simplifies the dev experience / debt on teams that are less DevOps/CI oriented.

The client won't have access to netlify's admin and for that we don't want them to be concerned with netlify at all.

Our netlify setup is sort of "proven and tested" and thats part of our proposition. So we want to make sure we have as much control as possible on the TOML so that key important featues are configured correctly and no offending plugins or config break builds (causing unnecessary issues). The client will manage a subset of netlify features (trigger deploys, logs etc) via our dashboard.

And finally, it helps us hide away some of our secret sauce.

Re: Installing plugins via the UI.
Do you guys support private build plugins via the UI or do they need to be public? - And if yes, could these plugins be added via the netlify CLI / API when we spin a new site?. We could be ok with just using defaults inputs tbh.

P.S - I'm def going to try your new feature to try and set base, build and functions settings. Setting these should be supported right?

[build]
  command = "yarn build"
  publish = "out_publish"
  functions = "out_functions"
  environment = { NETLIFY_USE_YARN = 'true', YARN_VERSION = '1.22.10' }

[functions]
  node_bundler = "esbuild"
  external_node_modules = [
    "caniuse-lite",
    "safer-buffer",
    "next/dist/compiled/@ampproject/toolbox-optimizer",
    "swiper"
  ]
  # since @netlify/plugin-nextjs@3.4.2:
  # we can include paths include their files in all nextjs/netlify
  # lambda functions, so that fs.readFile(...xyz.json) works
  included_files = [
    "public/json/collections/*.json",
    "public/json/products/item/*.json",
    "public/json/products/*.json",
    "public/json/*.json"
  ]

[dev]
  functions = "out_publish"
  publish   = "out_publish"
  framework = "#static"

[ehmicky]

Thanks for explaining your use case, this helps a lot and makes lots of sense.

Small side note: at a higher-level, I am wondering if account-level build settings would help out? Those would need to cover more Netlify configuration properties (for example, functions is not currently modifiable in the UI). Also, I'd assume most people would want account-level configuration to have the lowest priority (i.e. being just default values), while you would want the opposite. Regardless, this might an interesting idea.

Re: Installing plugins via the UI.
Do you guys support private build plugins via the UI or do they need to be public? - And if yes, could these plugins be added via the netlify CLI / API when we spin a new site?. We could be ok with just using defaults inputs tbh.

The Plugins directory in the UI is currently only meant as a source for curated, public Build plugins. Private plugins need to be installed in package.json instead.

P.S - I'm def going to try your new feature to try and set base, build and functions settings. Setting these should be supported right?

• Setting build.command, build.functions, build.environment, functions.node_bundler, functions.external_node_modules, functions.included_files is supported.
• Setting build.publish is not supported yet, but is currently being worked on.
• Setting dev.* is not supported at the moment.
• Setting build.base is not supported. The base directory determines the current directory used through the whole build, including for plugins themselves, which makes it hard to modify in the middle of the build.

Based on this, I am wondering whether this feature might not fit your use case. The feature's main use case is for plugins to modify specific configuration properties, not whole netlify.toml, and you might find yourself limited by the sets of configuration properties not supported yet.

So, the goal is to not have a TOML tracked in the repo.

Overall, it feels like you need a high level of control and abstraction over the Netlify configuration to provide with "managed Netlify" setups. From this angle, I am wondering if any solution that changes the configuration while the build is ongoing might be too limited for your use case. Instead, it does feel like you might actually need to be able to provide with a sets of Netlify configuration properties before the build even starts, which is what your initial idea (adding netlify.toml during postinstall) was gearing towards.

Would "shareable configurations" be a possible solution? Kindof like ESLint/Prettier shared configuration files. Link to feature request.

The main challenge here might be for this netlify.toml to remain private and hidden from users.

Are all the sites using the same account?

[juanpprieto]

Thanks to you @ehmicky. This is great.

Are all the sites using the same account?

Yes, we will provision all the sites through our partner account. - In the future, we may need to support clients providing their own netlify account (but not a priority for year one)

account-level build settings would help out

I think this would be ideal. A bit like the "Shared Env Vars". It would be great if we could upload a global netlify.toml via the CLI or API. Specially if it supported private plugins 🔥

Would "shareable configurations" be a possible solution? Kindof like ESLint/Prettier shared configuration files.

Yes, in the surface this feels like a good solution, but it would require adding more files to the starter (which again an enduser shouldnt be concerned with in this use case).

Agree that the current config feature may aid in ensuring certain fields configs (that are supported like functions), but it doesn't solve the bigger abstraction which is the [[plugins]] and the actual tracked file itself.

[ehmicky]

Thanks for the additional feedback.

The setup you are describing seems key to being able to provide any managed Netlify agency product offering. This definitely is a must-have from an agency standpoint.

One thing you would also want, I assume, is the ability to modify the configuration of all existing sites at once. From that angle, having account-level build settings (like "Shared Env Vars" as you mention) seems one possible way to go.

Private plugins might be another challenge since those are managed in package.json, not in netlify.toml nor in the UI build settings at the moment.

I have been reaching out to our product manager @verythorough and support engineer @kaganjd to see if they have additional insights into solving your use case.

[juanpprieto]

The setup you are describing seems key to being able to provide any managed Netlify agency product offering. This definitely is a must-have from an agency standpoint.

👀 💯 + Plus startups building a "starter-driven" offering on top of Netlify.

One thing you would also want, I assume, is the ability to modify the configuration of all existing sites at once. From that angle, having account-level build settings (like "Shared Env Vars" as you mention) seems one possible way to go.

Yes, this was the plan with the node --eval "require('@myorg/netlify-toml')". i.e so we could control config versions via this "netlify-toml" package.

Thanks once again for the long chat and assistance . One of the many reasons we chosen Netlify.

[ehmicky]

Yes, this was the plan with the node --eval "require('@myorg/netlify-toml')". i.e so we could control config versions via this "netlify-toml" package.

I am wondering whether your idea might share some concepts with the "shareable configurations" I mentioned above (the linked issue might not be representative anymore since it is quite dated). The basic idea would be similar to ESLint or Prettier shared configurations. For example, something along the lines of: to use your managed Netlify services, users are required to add a netlify.toml file with a single line:

extends = "@agency/netlify-toml"

@agency/netlify-toml could be a private Node module with a top-level netlify.toml.

This is not ideal since this requires a one-time setup action from your users for each of their site ("adding more files to the starter" as you mentioned above). However, this solution might fit several of your other requirements.

What are your thoughts on this?

[juanpprieto]

Yeah, I think this could be def a good stop gap. It would be even better if extends could be setup via CLI / UI. That could eliminate the file all together. :pepe: