Engine 502 and seems not working

[mdepedrof]

Describe the bug
few months ago I set out to deploy the Azure IPAM solution for a customer where I was deploying a LandingZone. As I was using Terraform I could not use the "official" deployment system that you have published so I tried to perform in terraform each of the steps that are done in powershell. Finally I managed to do it and I can say that it was a success because I could enter the panel, configure 2 spaces, a couple of blocks and add the vnets. I also used a Terraform module for AzureIpam that makes the reservations in the specified space. With all this context, I want to indicate that the deployed solution was working.

However, a couple of days ago I had to make changes to the registry which forced me to recompile the images and upload them again (it was not an apparent problem since there seemed to be no changes). However, since then, the application does not work. When you log into the web, it authenticates you and gets stuck on the screen with the spinning circle. Analyzing the endpoints that fail, I see that the endpoint /api/users/me?expand=true returns a 502.

I have looked at the logs, and I have seen that the container engine shows some error messages related to a module that is not found.

2024-02-19T16:39:58.877807185Z File "/ipam/app/main.py", line 14, in <module>
2024-02-19T16:39:58.877811785Z from app.routers import (
2024-02-19T16:39:58.877907087Z File "/ipam/app/routers/azure.py", line 43, in <module>
2024-02-19T16:39:58.877939087Z from app.logs.logs import ipam_logger as logger
2024-02-19T16:39:58.878033288Z ModuleNotFoundError: No module named 'app.logs'
2024-02-19T16:39:59.015812913Z Unclosed client session
2024-02-19T16:39:59.015834513Z client_session: <aiohttp.client.ClientSession object at 0x7c19d65a3a90>

I have read the documentation and I have modified the cosmos database with the option "DisableKeyBasedMetadataWriteAccess" in case this is the cause but the problem persists.

To Reproduce
Steps to reproduce the behavior:
I have deployed docker compose with 3 container (prod version) in a linux AppService with cosmosDB

I attach some screenshots and the docker logs where you can see the error I am talking about:

2024_02_19_ln0xsdlwk000003_nginx-proxy_docker.log
2024_02_19_ln0xsdlwk000003_ipam-engine_docker.log
2024_02_19_ln0xsdlwk000003_ipam-ui_docker.log
2024_02_19_ln0xsdlwk000003_docker.log

Screenshots

Note: if you nedd, i can try to share with you the TF files if you consider that could be intersting for others

[DCMattyG]

Hi @crower19, I'm so sorry you are running into this issue, but I'm happy to help!

My first question is did you pull the latest code from the main branch of the Azure IPAM GitHub repo before you recompiled your containers?

The reason this is important is that over the last few months, I've addressed many breaking changes in packages like FastAPI and Pydantic and if you have an older version of the code without those updates your containers could be building without those fixes and could explain the crashing that you are seeing.

So if you're copy of the codebase is older, I'd suggest pulling the latest code down and rebuilding your containers from that, otherwise if you're already running from the latest codebase we can start exploring some other possibilities together.

[mdepedrof]

Hi!!! thanks for your quickly answer. I downloaded the repo and build the docker images from the engine, lb and ui folders. I have check and the branch is main and its on latest commit.

In order to discard failures on my build automated process, I have clone the repo on new folder and then i run this command inside of the folders engine, lb and ui:

az acr build  --registry “xxxxxx"  --image "ipam-engine:latest"  --file "Dockerfile.deb" .

[mdepedrof]

Any idea?

thanks!

[DCMattyG]

Hi @crower19, it's really hard to determine what is going on here given that I'm not seeing this in any of my environments.

Could we perhaps have a live call to work through this together sometime this week or next?

Please drop me an email at Matthew.Garrett@microsoft.com and let's setup some time to figure out what's going on here.

[mdepedrof]

Hi! @DCMattyG. Reviewing the latest changes in the repo, I have seen that you no longer use Docker compose in the WebApp but you use a docker image directly. By applying these changes in Terraform, I was able to make it work again.

resource "azurerm_linux_web_app" "ipam" {
  count = local.enable_ipam ? 1 : 0

  name                            = replace(module.naming.compute_app_service_plan, "asp-management", "asp-ipam")
  location                        = azurerm_resource_group.this["rg-ipam"].location
  resource_group_name             = azurerm_resource_group.this["rg-ipam"].name
  service_plan_id                 = azurerm_service_plan.ipam[0].id
  key_vault_reference_identity_id = azurerm_user_assigned_identity.ipam[0].id

  https_only = true

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.ipam[0].id]

  }
  site_config {
    container_registry_use_managed_identity       = true
    container_registry_managed_identity_client_id = azurerm_user_assigned_identity.ipam[0].client_id
    always_on                                     = true
    app_command_line                              = "init.sh 8000"
    health_check_path                             = "/api/status"

    application_stack {
      docker_registry_url = "https://azureipam.azurecr.io"
      docker_image_name   = "ipam:latest"
    }
  }

  app_settings = {
    APPLICATIONINSIGHTS_CONNECTION_STRING           = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.ipam[0].vault_uri}secrets/APPINSIGHT-CONN-STRING/)"
    APPINSIGHTS_PROFILERFEATURE_VERSION             = "1.0.0"
    APPINSIGHTS_SNAPSHOTFEATURE_VERSION             = ""
    APPLICATIONINSIGHTS_CONFIGURATION_CONTENT       = ""
    ApplicationInsightsAgent_EXTENSION_VERSION      = "~3"
    DiagnosticServices_EXTENSION_VERSION            = "~3"
    InstrumentationEngine_EXTENSION_VERSION         = "disabled"
    SnapshotDebugger_EXTENSION_VERSION              = "disabled"
    XDT_MicrosoftApplicationInsights_BaseExtensions = "disabled"
    XDT_MicrosoftApplicationInsights_Mode           = "recommended"
    XDT_MicrosoftApplicationInsights_PreemptSdk     = "disabled"
    AZURE_ENV                                       = "AZURE_PUBLIC"
    COSMOS_URL                                      = azurerm_cosmosdb_account.ipam[0].endpoint
    COSMOS_KEY                                      = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.ipam[0].vault_uri}secrets/COSMOS-KEY/)"
    DATABASE_NAME                                   = azurerm_cosmosdb_sql_database.ipam[0].name
    CONTAINER_NAME                                  = azurerm_cosmosdb_sql_container.ipam[0].name
    UI_APP_ID                                       = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.ipam[0].vault_uri}secrets/UI-ID/)"
    ENGINE_APP_ID                                   = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.ipam[0].vault_uri}secrets/ENGINE-ID/)"
    ENGINE_APP_SECRET                               = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.ipam[0].vault_uri}secrets/ENGINE-SECRET/)"
    TENANT_ID                                       = "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.ipam[0].vault_uri}secrets/TENANT-ID/)"
    KEYVAULT_URL                                    = azurerm_key_vault.ipam[0].vault_uri
    WEBSITE_ENABLE_SYNC_UPDATE_SITE                 = "true"
  }

  logs {
    detailed_error_messages = true
    failed_request_tracing  = false

    application_logs {
      file_system_level = "Information"
    }

    http_logs {
      file_system {
        retention_in_days = 7
        retention_in_mb   = 35
      }
    }
  }
}

[DCMattyG]

Great news @crower19, and thank you so much for sharing the Terraform. I'm sure others will find it very useful in the future!

I'll go ahead and close this issue now, but please feel free to reach out with any other needs you may have. Happy to help however we can!