[Feature] Read environment variables for datasources values

[Nikhil-Nandagopal]

Summary

A user suggested an option to read datasource values from environment variables. This can help instance replication across environments

Front conversations

[0vint0]

Hello,
I've just started using AppSmith and want to thank you for great solution to create a BackOffice application 👍 .
I see using environment variables would be a great feature for this product. Let say I have 2 environments dev and prod
I would like to develop my AppSmith app in dev test it and only after export and import in prod, at the moment using different data-sources for dev and prod it implies for each time manual configuration of Db connections, API etc.
Having environment variables should simplify this, in that way we can define them either on docker level or in K8 configmap+deployment file
Ex how I see it should look like in configuration of DataSource

Thank you.

[raphaeltm]

This would be really helpful for us. I like @0vint0's solution to the problem. I think it would also be practical to be able to define these things either in the GUI or at the server level. For example, I could see us setting up an organization per environment (staging/prod/other) and defining variables per organization. I'm thinking of how Auth0, for example, lets you define variables per tenant.

I would personally envision AppSmith defaulting to the runtime environment variable, but being able to override that value with values specified at the organization level.

[dreinon]

Hi! Any updates with this? I have a login page that has a login button that redirects to a url that is different in dev and in prod. Since it is not a query but a raw text, I don't have a way to use git sync with this, so I cannot release the login feature to production. Any workarounds?

Thanks 🙂

[raphaeltm]

I would also love to know if there are any updates on this 🙂 Like @dreinon, it's also preventing me from making use of git sync in some cases. In one case we need to change the endpoint for a curl request between environments. Right now we've written some really awkward code to check the url and change the endpoint based on the current app id. It's.... far from ideal 😅

@dreinon the url thing might be something you could try as a temporary solution? We do a conditional which checks the current url to see if it's dev/staging/prod based on the app id, which is... awkward and not stable (especially since there are no "global" variables, so we have to duplicate this code in a few places), but it does the trick for now.

[Nikhil-Nandagopal]

@dreinon @raphaeltm you should be able to configure different environments using our git feature now. Once you import a git repo into a new app, it will prompt you to enter new credentials for each of the datasources. This can be used to configure different apps on the same repo to point to different environments. You can then create a staging branch on one app, commit your changes, merge it to a prod branch and then pull those changes onto the production app

[dreinon]

@Nikhil-Nandagopal right, this works really well for datasources, but in my case, since it is a url I use in the navigateTo function, it is not a datasource, it's a simple string, so there is no easy way to use different ones depending on the environment. If there was a URL readonly property for queries we could read, a workaround would be to add the url you want to redirect to as a query, but that's not available either.

[Nikhil-Nandagopal]

@dreinon ah got it! You can maintain the string in the JSObject and conditionally switch it based on the URL of the app. i.e
{{ appsmith.URL.queryParams.branch === "staging" ? "stagingURL" : "prodURL" }}
It's not the best solution right now but it should unblock you. You'll be happy to know that we're actively working on a holistic solution for this right now!

[dreinon]

@Nikhil-Nandagopal thanks! Yeah, as @raphaeltm said, conditionally checking for the app id in the url should do the trick for now.

[raphaeltm]

@Nikhil-Nandagopal am I understanding from your response that if we connect to a repo AppSmith will add a branch query param? Or is that something you're suggesting we add manually? If it's the former, and it's immutable, that will solve so many problems for me. If it's manual, then it doesn't help me much, since it's pretty much equivalent to our current approach.

On the subject, I remember talking to someone from AppSmith about global scoped JS Objects at one point (as opposed to per page). That would also essentially solve the problem, since it would be easy to switch a param in one place, potentially still based on the url.

[rhuanbarreto]

There's a point people didn't consider in this feature. This is very important because we don't want the app developers to have access to some secret values. And we need those secret values to be deployed preferably as Infra as Code (Terraform or Pulumi) and transferred to appsmith as environment variables so no developer has access to those values. Examples are user passwords for databases, client secrets for oauth authentication in datasources, among some configuration across dev/prod instances. I hope this feature could accomplish this.

[talo-sgh]

I wish I could do that

[Nikhil-Nandagopal]

@rhuanbarreto @talo-sgh the ask makes sense and we're going to build this out. I'm doing a little user research to cover all the cases properly. If you're up for a conversation that can help get this built better, this is my Calendly.
https://calendly.com/appsmith-nikhil/30min

[rhuanbarreto]

Booked!

[matmunn]

Did anything shake out of your chat? I have a similar use case as @rhuanbarreto

Though more specifically I wish to use environment variable values in JS objects.

[Nikhil-Nandagopal]

@matmunn we have this planned for Q2. We're not thinking of environment variables in JSObjects because environment variables are typically secrets for the datasource. Is there a strong need you have for this? You can just define these variables in your JSObject. If you are concerned about reusabilty, we're launching modules soon to solve that

[matmunn]

I'm using a third party JS library inside a JS object for interacting with an API - namely the Auth0 management API. If I embed credentials then there will be Auth0 secrets contained inside my git repository for my app.

[Nikhil-Nandagopal]

@matmunn Thanks! That's an interesting use case though I'm not sure how secure client-side env variables are for authentication because any app user can inspect the network requests and get the secret. I recommend using our SSO option for authentication instead or calling the Auth0 API using the REST API datasource. This will let you configure these credentials on the server and ensure the client never has access to them.

[joaocc]

Hi
A complement to this thread on automated initialization would be to configure the app from a config file (mounted into a well-known location).
This way we could spin up a new server completely configured without manual intervention.
Not being able to automate deployment flows is becoming a blocker for deployment on our side.
Thanks

[nivivive]

i'd love if we could have the appsmith ec2 instance org-level settings (not just the datasource attributes, but also users/roles/environmental-level settings) completely available via terraform to make it easier to spin up multiple self-hosted instances if we do an aws account per environment type of setup