Grap SVG error and extreme network utilization from console

[timbrigham-oc]

Describe the bug
While viewing operations after signing into version 5.0.0 of the console as a red user, a 20 Mbps or higher connection is required between the workstation and the console.

To Reproduce

1. Open the console. In my case this is a remote host at Azure, and protected by the SSL plugin.
2. Navigate to the "Operations" page.
3. Select an operation.
   The network view in Chrome shows alternating "summary" statements of 226 bytes (500 status code) and "operations" API calls listed (200 status code), each about 4.5 MB.
   Concurrently 500 status errors will show in Chrome's error console, with a summary of "Request content was evicted from inspector cache"

I dug into the 500 error since I've fixed handler issues in the past. It's a GET request to $baseURL/api/v2/operations/summary with a response body of
500 Internal Server Error Server got itself in trouble

Concurrently a message displays on screen, "Could not get operation graph data. see console for more information".
This corresponds to the graph svg not being generated.
However there is nothing present on the console, even increasing the logging level to debug.

There is no additional visible change on the screen when the data is continually loading.

I spent some time looking into the API endpoint where this is occurring, get_operations_summary.

The code was dying on the statements to

op['agents'] = self._api_manager.get_agents(op)
op['hosts'] = await self._api_manager.get_hosts(op)

I wrapped this with try / except blocks on my instance and it seems (?) to help.

            try:
              op['agents'] = self._api_manager.get_agents(op)
            except:
              print( "unable to set agents data" )
            try:
              op['hosts'] = await self._api_manager.get_hosts(op)
            except:
              print( "unable to set hosts data" )

It does at least result in the error banner being cleared and the svg is generated now.

Unfortunately that didn't clear up the repeating calls or the overall usage bandwidth being used, which was mostly from the options endpoint. That appears to be something upstream, maybe in the JS for the magma assets itself.

[timbrigham-oc]

Ok, some additional data. The repeated calls to the operations and summary API endpoints only happen if you navigate to the operations page, then select an operation off the dropdown menu. If you delete a record or otherwise don't have one already selected the constant reloading doesn't occur.

I've also cut the amount of data it's trying to pull back every time from the operations endpoint somewhat by culling old records I don't need any longer.

[timbrigham-oc]

Yeah, the overall bandwidth requirement was due in large part to the amount of data returned from the api/v2/operations endpoint combined with a crazy fast polling period. Since I'm still testing this out I had hundreds of test iterations, some of which with hundreds of individual lines. It was like 10 megabytes each time it gets called and easily a hundred times in a minute.

I used some PS scripting to bulk remove (what happened to being able to kill YML files??) that makes the console usable again.

$API_KEY = "xxxxxxxxxxxxxxxxxxxx"
$uri = "https://xxxxxxxxxxxxxxxxxxxxxxxx/api/v2/operations"
$headers = @{
    KEY = $API_KEY
}
$operations = Invoke-RestMethod -Uri $uri -Method GET -Headers $headers

foreach ( $operation in $operations ) {
    # if the operation name field contains the word "test" then remove it 
    #if ($operation.name -match "test") {
    write-output "Removing $($operation.name)"
    $opToRemove = $operation.id 
    $uri = "https://xxxxxxxxxxxxxx/api/v2/operations/$opToRemove"
    Invoke-RestMethod -Uri $uri -Method DELETE -Headers $headers
    write-output ==========
    #}
    #$operation.id 
}

[timbrigham-oc]

There's an issue with the operations API endpoint returning a ton of data and having a fast polling cycle in the current Magma release.. I've addressed it on my local instance in MAGMA#51. Long term fix should be a process that only polls changes IMO.