Data from HTTP POST request does not make it to the database

[diastoleus]

Running an issue from discussion #91 to ground. I did not have all of the facts at the time, but I think I have a pretty good picture of what is going on now:

• Clients are running properly.

• Clients are pushing timeline updates properly.

• API server is receiving HTTP POST requests from clients, verified through packet capture.

• ghost-api container logs an error 13 when recieving the POST request. It references dotnet errors involving null values in variables that cannot be null. Examining the HTTP POST request payload revealed that the data had multiple null values (perhaps that's what it is referencing?). Timeline update never makes it to the database.

I apologize for the lack of granularity; when I am back on network in a few hours I will post exact log output/packet data to assist in troubleshooting.

[sei-dupdyke]

Hi, hmm, could you post that payload with the null values in it by chance? I will try to recreate here. Please standby...

[diastoleus]

Hi, hmm, could you post that payload with the null values in it by chance? I will try to recreate here. Please standby...

Absolutely, but I won't be back in the office for 10 hours or so. I'll data dump everything relevant as soon as I'm back in.

[diastoleus]

HTTP stream from agent to API server

GET /api/clientid HTTP/1.1
ghosts-name: hostname
ghosts-fqdn: hostname.domain
ghosts-host: hostname
ghosts-domain: domain
ghosts-resolvedhost: hostname.domain
ghosts-ip: Client_IP
ghosts-user: TG9jYWwgVXNlcg==
ghosts-version: 6.0.2.6
User-Agent: Ghosts Client
Host: API_Server_IP:Port
Connection: Keep-Alive

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Jun 2022 06:27:10 GMT
Server: Kestrel
Transfer-Encoding: chunked

"Machine_ID"GET /api/clientupdates HTTP/1.1
ghosts-id:
ghosts-name: hostname
ghosts-fqdn: hostname.domain
ghosts-host: hostname
ghosts-domain: domain
ghosts-resolvedhost: hostname.domain
ghosts-ip: Client_IP
ghosts-user: TG9jYWwgVXNlcg==
ghosts-version: 6.0.2.6
User-Agent: Ghosts Client
Host: API_Server_IP:Port

HTTP/1.1 500 Internal Server Error
Content-Length: 0
Date: Tue, 28 Jun 2022 06:27:10 GMT
Server: Kestrel

GET /api/clientid HTTP/1.1
ghosts-name: hostname
ghosts-fqdn: hostname.domain
ghosts-host: hostname
ghosts-domain: domain
ghosts-resolvedhost: hostname.domain
ghosts-ip: Client_IP
ghosts-user: TG9jYWwgVXNlcg==
ghosts-version: 6.0.2.6
User-Agent: Ghosts Client
Host: API_Server_IP:Port

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Jun 2022 06:27:10 GMT
Server: Kestrel
Transfer-Encoding: chunked

"Machine_ID"POST /api/clientresults HTTP/1.1
ghosts-id:
ghosts-name: hostname
ghosts-fqdn: hostname.domain
ghosts-host: hostname
ghosts-domain: domain
ghosts-resolvedhost: hostname.domain
ghosts-ip: Client_IP
ghosts-user: TG9jYWwgVXNlcg==
ghosts-version: 6.0.2.6
Content-Type: application/json
User-Agent: Ghosts Client
Host: API_Server_IP:Port
Content-Length: 1749

{"Log":"TIMELINE|6/8/2022 11:35:38 AM|{"Handler":"Command","Command":"explorer \\\\nec-file\\schedule","CommandArg":""}\r\nTIMELINE|6/8/2022 11:35:38 AM|{"Handler":"Command","Command":"explorer \\\\nec-file\\reports","CommandArg":""}\r\nTIMELINE|6/8/2022 11:35:38 AM|{"Handler":"Command","Command":"explorer \\\\nec-file\\base","CommandArg":""}\r\nTIMELINE|6/8/2022 11:35:39 AM|{"Handler":"Command","Command":"exit","CommandArg":""}\r\nTIMELINE|6/8/2022 11:35:39 AM|{"Handler":"Command","Command":"exit","CommandArg":""}\r\nTIMELINE|6/8/2022 11:35:39 AM|{"Handler":"Command","Command":"exit","CommandArg":""}\r\nTIMELINE|6/8/2022 11:35:40 AM|{"Handler":"BrowserFirefox","Command":"random","CommandArg":"{\"Uri\":\"http://httpbin.org/post\\\",\\\"Category\\\":\\\"cat1\\\",\\\"Method\\\":\\\"POST\\\",\\\"Headers\\\":{\\\"1\\\":\\\"a\\\",\\\"2\\\":\\\"b\\\"},\\\"FormValues\\\":{\\\"1\\\":\\\"a\\\",\\\"2\\\":\\\"b\\\"},\\\"Body\\\":null}\"}\r\nTIMELINE|6/8/2022 11:40:32 AM|{"Handler":"Command","Command":"explorer \\\\nec-file\\base","CommandArg":""}\r\nTIMELINE|6/8/2022 11:40:32 AM|{"Handler":"Command","Command":"ipconfig","CommandArg":""}\r\nTIMELINE|6/8/2022 11:40:33 AM|{"Handler":"Command","Command":"exit","CommandArg":""}\r\nTIMELINE|6/8/2022 11:40:33 AM|{"Handler":"Command","Command":"exit","CommandArg":""}\r\nTIMELINE|6/8/2022 11:40:59 AM|{"Handler":"BrowserFirefox","Command":"random","CommandArg":"{\"Uri\":\"http://wikipedia.org\\\",\\\"Category\\\":null,\\\"Method\\\":\\\"GET\\\",\\\"Headers\\\":null,\\\"FormValues\\\":null,\\\"Body\\\":null}\"}\r\n"}HTTP/1.1 204 No Content
Date: Tue, 28 Jun 2022 06:27:10 GMT
Server: Kestrel

Ghost-API container log generated when the clientresults POST request hits the server

2022-06-27T12:22:48.102328695Z Hosting environment: Production
2022-06-27T12:22:48.102361948Z Content root path: /app
2022-06-27T12:22:48.102508611Z Now listening on: http://[::]:5000
2022-06-27T12:22:48.102524912Z Application started. Press Ctrl+C to shut down.
2022-06-27T12:58:16.544350529Z System.ArgumentNullException: Value cannot be null. (Parameter 'input')
2022-06-27T12:58:16.544392607Z at System.Text.RegularExpressions.ThrowHelper.ThrowArgumentNullException(ExceptionArgument arg)
2022-06-27T12:58:16.544398261Z at System.Text.RegularExpressions.Regex.IsMatch(String input)
2022-06-27T12:58:16.544402266Z at Ghosts.Api.Infrastructure.WebRequestReader.CheckIfBase64Encoded(String raw) in /app/Ghosts.Api/Infrastructure/WebRequestReader.cs:line 48
2022-06-27T12:58:16.544405915Z at Ghosts.Api.Infrastructure.WebRequestReader.GetMachine(HttpContext context) in /app/Ghosts.Api/Infrastructure/WebRequestReader.cs:line 17
2022-06-27T12:58:16.709832354Z {"EventId":13,"LogLevel":"Error","Category":"Microsoft.AspNetCore.Server.Kestrel","Message":"Connection id \u00220HMIO79MV9DRM\u0022, Request id \u00220HMIO79MV9DRM:00000002\u0022: An unhandled exception was thrown by the application.","Exception":"System.ArgumentNullException: Value cannot be null. (Parameter \u0027input\u0027) at System.Text.RegularExpressions.ThrowHelper.ThrowArgumentNullException(ExceptionArgument arg) at System.Text.RegularExpressions.Regex.IsMatch(String input) at Ghosts.Api.Infrastructure.WebRequestReader.CheckIfBase64Encoded(String raw) in /app/Ghosts.Api/Infrastructure/WebRequestReader.cs:line 48 at Ghosts.Api.Infrastructure.WebRequestReader.GetMachine(HttpContext context) in /app/Ghosts.Api/Infrastructure/WebRequestReader.cs:line 17 at ghosts.api.Controllers.ClientTimelineController.Index(String timeline, CancellationToken ct) in /app/Ghosts.Api/Controllers/ClientTimeline.cs:line 47 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003Eg__Awaited|12_0(ControllerActionInvoker invoker, ValueTask\u00601 actionResultValueTask) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextResourceFilter\u003Eg__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeAsync\u003Eg__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) at Microsoft.AspNetCore.Routing.EndpointMiddleware.\u003CInvoke\u003Eg__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication\u00601 application)","State":{"Message":"Connection id \u00220HMIO79MV9DRM\u0022, Request id \u00220HMIO79MV9DRM:00000002\u0022: An unhandled exception was thrown by the application.","ConnectionId":"0HMIO79MV9DRM","TraceIdentifier":"0HMIO79MV9DRM:00000002","{OriginalFormat}":"Connection id \u0022{ConnectionId}\u0022, Request id \u0022{TraceIdentifier}\u0022: An unhandled exception was thrown by the application."}}

[diastoleus]

Of note, "ghosts-id" was not omitted; there was no value in that field.

Also our agent is configured to run excel, powerpoint etc, but the update logs aren't generating on the host and are in turn not being pushed to the API server.

[sei-dupdyke]

Wow, thanks for all the log info, makes troubleshooting so much easier—thanks!

The first call to GET /api/clientid should return a uuid for that client based on its hostname/fqdn. Here is what it should look like in postman:

That uuid is stored by the client in ./instance/id.json and gets sent with every other client request.

The second call for a clientupdate to GET /api/clientupdates will fail without a uuid value in the ghosts-id header, otherwise it will have an update or return 404 for no updates.

The call to POST /api/clientresults will also fail without a uuid value in the ghosts-id header. This is why the client logs are not getting off-loaded to the server

So I think our root issue here is that the id is not getting stored by the client for some reason. I have not seen this before.

Do you have an instance folder? Is there an id.json file in there? If not — as a first step — if you create a file ./instance/id.json and put any uuid in that file (with no quotes or anything else), everything else should start working as normal. Here is an example:

I'll keep investigating on this side. Sorry the API message is not more clear as to the id being missing. I'll fix that.

[diastoleus]

Well, I have some not so great news. We do have an instance folder, and there is an id.json file with what appears to be a valid UUID.

[sei-dupdyke]

Flush the logs and restart ghosts. Anything in app.log?