Django environment classification defaulting to dev

[mitchdawson1982]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

^2.0.1

Steps to Reproduce

We have a Django application running across three environments (test, dev and preprod), each environment has an ENV variable configured with the respective value. This variable is used as the environment argument in the sdk init.

We noticed that events were being received in Sentry with an environment of dev only, i.e. test or preprod were never used despite being configured correctly. When validating this behaviour locally I have also found that periodically alerts will come in with the environment as dev despite an entirely different value being provided. I have further noticed that when this occurs, the release value is also different from a working example.

# Sentry Configuration
sentry_sdk.init(
    dsn=os.environ.get("SENTRY_DSN"),
    enable_tracing=True,
    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for performance monitoring.
    traces_sample_rate=1.0,
    # Set profiles_sample_rate to 1.0 to profile 100%
    # of sampled transactions.
    # We recommend adjusting this value in production.
    profiles_sample_rate=1.0,
    environment=ENV or "local",
)

Expected Result

When ENV=new-env1 received alerts should show the environment as new-env1

Actual Result

When ENV=new-env1 received alerts periodically show the environment as dev

[szokeasaurusrex]

Hi @mitchdawson1982, thank you for reporting this problem. Could you provide us a link to one of the error events where the environment was set to dev, even though the event originated from a non-dev environment to help us investigate?

[Angelodaniel]

Hey @szokeasaurusrex -> Shared the link directly with you

[mitchdawson1982]

Hi @szokeasaurusrex Sure here is one of the links

I have noticed that the trace details are populated with much more information for a working example.

[sentrivana]

@mitchdawson1982 Could you try using the SENTRY_ENVIRONMENT env var instead? If you set SENTRY_ENVIRONMENT, it'll be used automatically, so you could remove it from the init. See also here.

As far as I can tell the SDK will never set the environment to dev by itself. This is the only place I found where we set it to something (if it's not there), and we default to production. So my hunch is this might be originating somewhere else. Maybe something external, like a tool, is setting the ENV var, which is why I'm wondering if the issue goes away after switching to something more unique like SENTRY_ENVIRONMENT.

Since the faulty events also have a different release though, is it at all possible the dev events you're seeing are coming from a different place entirely? (Automated tests, etc.) Is the non-hash release something that you'd expect seeing in your local environment?

[mitchdawson1982]

@mitchdawson1982 Could you try using the SENTRY_ENVIRONMENT env var instead? If you set SENTRY_ENVIRONMENT, it'll be used automatically, so you could remove it from the init. See also here.

As far as I can tell the SDK will never set the environment to dev by itself. This is the only place I found where we set it to something (if it's not there), and we default to production. So my hunch is this might be originating somewhere else. Maybe something external, like a tool, is setting the ENV var, which is why I'm wondering if the issue goes away after switching to something more unique like SENTRY_ENVIRONMENT.

Since the faulty events also have a different release though, is it at all possible the dev events you're seeing are coming from a different place entirely? (Automated tests, etc.) Is the non-hash release something that you'd expect seeing in your local environment?

Thanks for the suggestion I will try this out when I get time. In terms of where the events are originating from, I cant envisage them coming from anywhere else. We have tests but these are manually invoked, the scenario I have described is where the app is being run normally.

[mitchdawson1982]

Hi @sentrivana I have performed further testing this morning with and without the SENTRY_ENVIRONMENT variable. To clarify I am running this application utilising the current main branch on my local Macbook.

I have been thinking about the rationale provided for this behaviour and would like to clarify some of the points.
On the tests where we have populated the ENVvariable and sentry_sdk.init(environment=ENV or "local"), I have in nearly all cases printed out the value of ENV to the terminal after sentry_sdk.init(environment=ENV or "local") in settings.py to validate the correct value is passed as expected. Where I have done so this has always been consistent for both correct and incorrect alerts and therefore gives me a high level of confidence that the right value has always been passed to the sdk environment argument. I don't see any likelihood of our code being able to manipulate the environment argument within sentry after the sdk init has been called.

Where correct alerts are seen the release hash matches the correct commit hash i.e. this alert has the latest commit hash in main. However for incorrect alerts we see 0.12.1.5. The only correlation I can find with this number is the version of the acryl-datahub package we have installed. Which is a dependency of a library we have written in this application to talk to the backend datahub application.

Results

I have tested several with the SENTRY_ENVIRONMENT variable set to sent-env with the environment argument removed from the sdk init. In all cases the alert environmental value was persistent but there was variance in the trace details and the release names. Please see below.

[antonpirker]

In summary, when the wrong environment is set also the release is wrong. My hunch is that someone else is running the project that has set ENV to the wrong value.

And this other one that runs the project does not run it from a Git repository. How the Sentry Python SDK sets the release, if there is none specified in sentry_sdk.init(), isdescribed here (below the code snippet): https://docs.sentry.io/platforms/python/configuration/releases/#setting-a-release

Do you have the project running in Google App Engine, Azure, AWS, Heroku, Codebuild, Circle CI, GitLab CI/CD, or Travis CI?

[mitchdawson1982]

In summary, when the wrong environment is set also the release is wrong. My hunch is that someone else is running the project that has set ENV to the wrong value.

And this other one that runs the project does not run it from a Git repository. How the Sentry Python SDK sets the release, if there is none specified in sentry_sdk.init(), isdescribed here (below the code snippet): https://docs.sentry.io/platforms/python/configuration/releases/#setting-a-release

Do you have the project running in Google App Engine, Azure, AWS, Heroku, Codebuild, Circle CI, GitLab CI/CD, or Travis CI?

I'm not as yet convinced that the environment is being set wrong. On the tests where we have populated the ENV variable and sentry_sdk.init(environment=ENV or "local"), I have in nearly all cases printed out the value of ENV to the terminal after sentry_sdk.init(environment=ENV or "local") in settings.py to validate the correct value is passed as expected. Where I have done so this has always been consistent for both correct and incorrect alerts and therefore gives me a high level of confidence that the right value has always been passed to the sdk environment argument. Once this value has been passed to the sdk init, how could our code manipulate this value? I see this behaviour on local development laptops and three separate k8 pods.

[sentrivana]

Where correct alerts are seen the release hash matches the correct commit hash i.e. this alert has the latest commit hash in main. However for incorrect alerts we see 0.12.1.5. The only correlation I can find with this number is the version of the acryl-datahub package we have installed. Which is a dependency of a library we have written in this application to talk to the backend datahub application.

Is it possible acryl-datahub itself is using Sentry? I don't have any experience with it but there seem to be some actual hits for sentry in the repo that I got to via their PyPI. This line seems especially interesting.

[mitchdawson1982]

Thanks, I am currently investigating this.

[sentrivana]

@mitchdawson1982 I see the linked MoJ issue is closed -- have you been able to resolve this or do you need any additional support from our end?

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[sentrivana]

Closing. Please reopen if not solved.