Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ BUG ] Returning empty Body ("body": {"message": "No content returned", "resources": []}) #1033

Closed
Kish26 opened this issue Aug 31, 2023 · 29 comments · Fixed by #1036
Closed

[ BUG ] Returning empty Body ("body": {"message": "No content returned", "resources": []}) #1033

Kish26 opened this issue Aug 31, 2023 · 29 comments · Fixed by #1036
Assignees
@jshcodes
Labels
bug 🐛 Something isn't working code samples Functioning sample code to get you started reports Reporting questions or issues

Comments

@Kish26
Copy link

Kish26 commented Aug 31, 2023 •
edited by jshcodes
Loading

I am using the below code to pull the information, it was working fine yesterday but it is not returning any content in json's body.

Can you please help ?

import json
from argparse import ArgumentParser, RawTextHelpFormatter
from falconpy import ReportExecutions
import os

def consume_arguments():
    """Consume our required command line arguments."""
    parser = ArgumentParser(description=__doc__, formatter_class=RawTextHelpFormatter)
    required = parser.add_argument_group("required_arguments")
    required.add_argument("-k", "--falcon_client_id",
                        help="CrowdStrike API Client ID",
                        required=True
                        )
    required.add_argument("-s", "--falcon_client_secret",
                        help="CrowdStrike API Client Secret",
                        required=True
                        )
    required.add_argument("-r", "--report", help="ID of the report to retrieve", required=True)
    return parser.parse_args()


def retrieve_report_executions(sdk: ReportExecutions, rptid: str):
    """Retrieve the list of execution IDs that match this report ID."""
    print(f"🔍 Searching for executions of {rptid}")
    execution_id_lookup = sdk.reports_executions_query(filter=f"scheduled_report_id:'{rptid}'")
    if not execution_id_lookup["status_code"] == 200:
        raise SystemExit("⛔ Unable to retrieve report executions from "
                         "the CrowdStrike API, check API key permissions."
                         )

    # Give the SDK back so we can feed our results to the next method easily
    return sdk, execution_id_lookup["body"]["resources"]


def get_report_execution_runs(sdk: ReportExecutions, id_list: list):
    """Retrieve the list of execution runs for each execution ID."""
    print(f"✅ Found {len(id_list)} executions of this report available.")
    # Retrieve the status of these IDs
    exec_status_lookup = sdk.report_executions_get(id_list)
    if not exec_status_lookup["status_code"] == 200:
        raise SystemExit("⛔ Unable to retrieve execution statuses from the CrowdStrike API.")
    print(f"⚠️  This execution has run {len(exec_status_lookup['body']['resources'])} times.")

    # Give the SDK back as well so we can easily feed it to our next method call
    return sdk, exec_status_lookup["body"]["resources"]


def process_executions(sdk: ReportExecutions, run_list: list):
    """Process the results of the executions, this solution only handles completed runs."""
    saved = 0
    for exec_status in run_list:
        status = exec_status["status"]
        exec_id = exec_status["id"]
        rpt_id = exec_status["scheduled_report_id"]
        if status.upper() == "DONE":
            report_detail = sdk.get_download(exec_id)
            if report_detail:
                if isinstance(report_detail, dict):
                    try:
                        with open(f"{rpt_id}_{exec_id}.rpt", "w", encoding="utf-8") as json_output:
                            json.dump(report_detail, json_output)
                        saved += 1
                        print(f"📥 {exec_id} successfully saved to {rpt_id}_{exec_id}.rpt")
                    except json.JSONDecodeError:
                        print(f"❗ Unable to decode results of report run {exec_id} for ")
                else:
                    with open(f"{rpt_id}_{exec_id}.rpt", "wb") as csv_output:
                        csv_output.write(report_detail)
                    saved += 1
            else:
                print(f"⛔ Unable to retrieve report for execution {exec_id} of {rpt_id}.")
        else:
            print(f"⏩ Skipping {exec_id} as not yet finished.")
    # Return back the number of successful saves
    return saved


if __name__ == "__main__":
    # Consume any provided command line arguments
    cmdline = consume_arguments()
    # Create an instance of the ReportExecutions Service Class
    falcon = ReportExecutions(client_id=os.getenv("CLIENT_ID"),
                              client_secret=cmdline.falcon_client_secret
                              )
    # Retrieve our report executions, and process them, saving any that
    # have completed successfully to individual files (JSON format).
    # Let's be fancy and leverage list expansion to provide arguments from
    # one method to the subsequent one. It's like inception for Python. ♜
    SUCCESSFUL = process_executions(
        *get_report_execution_runs(*retrieve_report_executions(falcon, cmdline.report))
        )
    # Inform the user of the result
    print(f"🏁 Retrieval complete, {SUCCESSFUL} report results were downloaded.")
@Kish26 Kish26 added the bug 🐛 Something isn't working label Aug 31, 2023
@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023

No investigation needed

@Kish26 Kish26 closed this as completed Aug 31, 2023
@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023

not returning any data as of today.....it was working until yesterday

@Kish26 Kish26 reopened this Aug 31, 2023
@Kish26 Kish26 closed this as completed Aug 31, 2023
@Kish26 Kish26 reopened this Aug 31, 2023
@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023

It looks like problem in JSON file format where it get downloaded from the schedule report.

@jshcodes jshcodes added code samples Functioning sample code to get you started reports Reporting questions or issues labels Aug 31, 2023
@jshcodes
Copy link
Member

Hi @Kish26 - I'm investigating this. Anything special about the report you're trying to retrieve? Is it configured for CSV or JSON format?

@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023 via email

@jshcodes
Copy link
Member

jshcodes commented Aug 31, 2023 •
edited
Loading

I think I've found the issue, but have another follow up question.

Which version of FalconPy are you using? Has this changed over the past day?

@jshcodes jshcodes changed the title [ BUG ] ...Returning empty Body ("body": {"message": "No content returned", "resources": []}) [ BUG ] Returning empty Body ("body": {"message": "No content returned", "resources": []}) Aug 31, 2023
@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023 via email

@jshcodes
Copy link
Member

what command I need to run to find a version please ? Thanks -Siva

If you've installed FalconPy for all users you should be able to see it with pip show requests (or pip3 show requests). If you're in a virtual environment, it would be something like pipenv graph.

You can also ask FalconPy by checking the value of the _VERSION constant.

from falconpy import _VERSION
print(_VERSION)

@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023 via email

@jshcodes jshcodes self-assigned this Aug 31, 2023
@cvjbrooks
Copy link

FWIW:

I am in a similar situation. I've been using falconpy without changes to my code for a long time, but today when i executed, it was only working about 25% of the time. The response did not contain any content so my code failed when looking for the meta key. The issue is intermittent and makes me lean towards an issue with the API but I am not positive.

I chatted with support to inquire about a status page to see on-going issues and after providing him with my API client ID, he said it looked like rate limiting was my issue. I was skeptical, since i've never had the issue before today but perhaps they've changed the limits. I am adding some debug lines to see if I can get more info next time it fails.

code that fails:

    while counter < total:
        result = falcon.query_devices_by_filter_scroll(
            sort="last_seen|asc",
            limit=max_rows,
            offset=offset_value)["body"]

crowdstrike-falconpy version 1.2.12

@jshcodes
Copy link
Member

jshcodes commented Aug 31, 2023 •
edited
Loading

We've found a programmatic fix for this issue (related to detecting the unusual payload return) that we can implement in the Result object. This is specific to how 1.3.0 handles this endpoint response.

We'll have the programmatic fix included as part of the 1.3.1 release posting this week.

@jshcodes
Copy link
Member

jshcodes commented Aug 31, 2023 •
edited
Loading

@Kish26 - Could you retest for me and let me know if you're still seeing this issue? I believe it should be resolved for versions prior to 1.3.0.

@cvjbrooks
Copy link

I am adding some debug lines to see if I can get more info next time it fails.

@cvjbrooks -

Are you able to update to FalconPy v1.3.0? Then you can take advantage of Debug Logging.

I enabled debug logging and tested at least 15 times and it worked each time, so whatever problem it was seems to be better now. No issues with rate limits or anything. 🤷 .

To correct an error in my original reply, I was already on 1.3.0 but mistakenly saw 1.2.12 in requirements.txt. I just switched to using pyproject.toml recently and had ^1.2.12 set. If it happens again, i'll post logs here if needed.

@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023

I just ran the report and same error message which is "body": {"message": "No content returned"", "resources": []}

@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023

Scheduled Report --> Host......payload still works.
image

@jshcodes
Copy link
Member

Which report type is still not working?

@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023

Installed patches and Vulnerabilities aren't working

@Kish26
Copy link
Author

Kish26 commented Aug 31, 2023

installed patches -->last ran successfully was 31st of July
Vulnerabilities --> last ran successfully was 30th of August

jshcodes added a commit that referenced this issue Sep 1, 2023
@jshcodes
Handle list response on report_executions_download_get. Closes #1033.
b5982ae
@jshcodes jshcodes mentioned this issue Sep 1, 2023
Merged
4 tasks
@Kish26
Copy link
Author

Kish26 commented Sep 1, 2023

Please let me know, if you want me to retest or upgraded to newer FalconPy version

jshcodes added a commit that referenced this issue Sep 1, 2023
@jshcodes
Handle list response on report_executions_download_get. Closes #1033.
a519f9f
jshcodes added a commit that referenced this issue Sep 1, 2023
@jshcodes
Handle list response on report_executions_download_get. Closes #1033.
9b12498
jshcodes added a commit that referenced this issue Sep 1, 2023
@jshcodes
Handle list response on report_executions_download_get. Closes #1033.
048a002
jshcodes added a commit that referenced this issue Sep 1, 2023
@jshcodes
Handle list response on report_executions_download_get. Closes #1033.
f6bd8ae
@jshcodes
Copy link
Member

jshcodes commented Sep 1, 2023

Please let me know, if you want me to retest or upgraded to newer FalconPy version

Hi @Kish26 -

I've tested using the 1.2.12 version to request variations of these reports, and cannot recreate the issue at this point. This may be related to the configuration of these two reports. (You could try triggering them manually to see if they run, or recreating them.) My test reports are very generic, with only filter.

Note: I do have a fix for the 1.3.x codebase that will be posting to the main branch today.

@Kish26
Copy link
Author

Kish26 commented Sep 1, 2023 via email •
edited by jshcodes
Loading

@jshcodes
Copy link
Member

jshcodes commented Sep 1, 2023 •
edited
Loading

That behavior makes me want to see the API response, which is harder to do in version 1.2.12. Perhaps getting you moved to the latest version is the right call here. 🤔

Version 1.3.1 is almost through unit testing and final reviews. This would give you the ability to use the new Debug Logging functionality, and it also contains the Result object fix. You could pull it early as there will be a bleeding edge release once the updated code merges to main. (Should merge today.)

I'll link you here to the pre-release announcement once it posts, the instructions for installing the bleeding edge package will be in there. (You'll install from the test PyPI index instead. Your code will not have to be changed.)

The product package release will happen on either Tuesday or Wednesday depending on when the merge happens.

jshcodes added a commit that referenced this issue Sep 1, 2023
@jshcodes
Handle list response on report_executions_download_get. Closes #1033.
f515d0a
@Kish26
Copy link
Author

Kish26 commented Sep 1, 2023 via email

jshcodes added a commit that referenced this issue Sep 1, 2023
@jshcodes
Handle list response on report_executions_download_get. Closes #1033.
895a77a
@jshcodes
Copy link
Member

jshcodes commented Sep 1, 2023

Hi @Kish26 -

Version 1.3.1 has merged. You can find details and instructions for installing the bleeding edge release here.

Let us know if you still have this issue afterwards (feel free to reopen this issue, or post a new one). 😄

@Kish26
Copy link
Author

Kish26 commented Sep 1, 2023

Thank you, this time i am getting status_code': 500

{'status_code': 500, 'headers': {}, 'body': {'errors': [{'message': "'ascii' codec can't decode byte 0xc2 in position 1508426: ordinal not in range(128)", 'code': 500}],
'resources': []}}

@jshcodes
Copy link
Member

jshcodes commented Sep 1, 2023

If you've performed the upgrade, you should be able to access Debug Logging. Here's a variation of the script above that turns it on. We're need to know what the RESULT payload looks like for the last call.

import json
import logging
from argparse import ArgumentParser, RawTextHelpFormatter
from falconpy import ReportExecutions
import os
logging.basicConfig(level=logging.DEBUG)

def consume_arguments():
    """Consume our required command line arguments."""
    parser = ArgumentParser(description=__doc__, formatter_class=RawTextHelpFormatter)
    required = parser.add_argument_group("required_arguments")
    required.add_argument("-k", "--falcon_client_id",
                        help="CrowdStrike API Client ID",
                        required=True
                        )
    required.add_argument("-s", "--falcon_client_secret",
                        help="CrowdStrike API Client Secret",
                        required=True
                        )
    required.add_argument("-r", "--report", help="ID of the report to retrieve", required=True)
    return parser.parse_args()


def retrieve_report_executions(sdk: ReportExecutions, rptid: str):
    """Retrieve the list of execution IDs that match this report ID."""
    print(f"🔍 Searching for executions of {rptid}")
    execution_id_lookup = sdk.reports_executions_query(filter=f"scheduled_report_id:'{rptid}'")
    if not execution_id_lookup["status_code"] == 200:
        raise SystemExit("⛔ Unable to retrieve report executions from "
                         "the CrowdStrike API, check API key permissions."
                         )

    # Give the SDK back so we can feed our results to the next method easily
    return sdk, execution_id_lookup["body"]["resources"]


def get_report_execution_runs(sdk: ReportExecutions, id_list: list):
    """Retrieve the list of execution runs for each execution ID."""
    print(f"✅ Found {len(id_list)} executions of this report available.")
    # Retrieve the status of these IDs
    exec_status_lookup = sdk.report_executions_get(id_list)
    if not exec_status_lookup["status_code"] == 200:
        raise SystemExit("⛔ Unable to retrieve execution statuses from the CrowdStrike API.")
    print(f"⚠️  This execution has run {len(exec_status_lookup['body']['resources'])} times.")

    # Give the SDK back as well so we can easily feed it to our next method call
    return sdk, exec_status_lookup["body"]["resources"]


def process_executions(sdk: ReportExecutions, run_list: list):
    """Process the results of the executions, this solution only handles completed runs."""
    saved = 0
    for exec_status in run_list:
        status = exec_status["status"]
        exec_id = exec_status["id"]
        rpt_id = exec_status["scheduled_report_id"]
        if status.upper() == "DONE":
            report_detail = sdk.get_download(exec_id)
            if report_detail:
                if isinstance(report_detail, dict):
                    try:
                        with open(f"{rpt_id}_{exec_id}.rpt", "w", encoding="utf-8") as json_output:
                            json.dump(report_detail, json_output)
                        saved += 1
                        print(f"📥 {exec_id} successfully saved to {rpt_id}_{exec_id}.rpt")
                    except json.JSONDecodeError:
                        print(f"❗ Unable to decode results of report run {exec_id} for ")
                else:
                    with open(f"{rpt_id}_{exec_id}.rpt", "wb") as csv_output:
                        csv_output.write(report_detail)
                    saved += 1
            else:
                print(f"⛔ Unable to retrieve report for execution {exec_id} of {rpt_id}.")
        else:
            print(f"⏩ Skipping {exec_id} as not yet finished.")
    # Return back the number of successful saves
    return saved


if __name__ == "__main__":
    # Consume any provided command line arguments
    cmdline = consume_arguments()
    # Create an instance of the ReportExecutions Service Class
    falcon = ReportExecutions(client_id=os.getenv("CLIENT_ID"),
                              client_secret=cmdline.falcon_client_secret,
                              debug=True
                              )
    # Retrieve our report executions, and process them, saving any that
    # have completed successfully to individual files (JSON format).
    # Let's be fancy and leverage list expansion to provide arguments from
    # one method to the subsequent one. It's like inception for Python. ♜
    SUCCESSFUL = process_executions(
        *get_report_execution_runs(*retrieve_report_executions(falcon, cmdline.report))
        )
    # Inform the user of the result
    print(f"🏁 Retrieval complete, {SUCCESSFUL} report results were downloaded.")

@Kish26
Copy link
Author

Kish26 commented Sep 1, 2023

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.crowdstrike.com:443
DEBUG:urllib3.connectionpool:https://api.crowdstrike.com:443 "POST /oauth2/token HTTP/1.1" 201 1244
🔍 Searching for executions of 6394ed9433254eb5b0bcadfcfbba8c93
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.crowdstrike.com:443
DEBUG:urllib3.connectionpool:https://api.crowdstrike.com:443 "GET /reports/queries/report-executions/v1?filter=scheduled_report_id%3A%276394ed9433254eb5b0bcadfcfbba8c93%27 HTTP/1.1" 200 221
✅ Found 1 executions of this report available.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.crowdstrike.com:443
DEBUG:urllib3.connectionpool:https://api.crowdstrike.com:443 "GET /reports/entities/report-executions/v1?ids=366995f0bbe24aaa925c0685d6eaea4b HTTP/1.1" 200 461
⚠️ This execution has run 1 times.
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.crowdstrike.com:443
DEBUG:urllib3.connectionpool:https://api.crowdstrike.com:443 "GET /reports/entities/report-executions-download/v1?ids=366995f0bbe24aaa925c0685d6eaea4b HTTP/1.1" 200 None
📥 366995f0bbe24aaa925c0685d6eaea4b successfully saved to 6394ed9433254eb5b0bcadfcfbba8c93_366995f0bbe24aaa925c0685d6eaea4b.rpt
🏁 Retrieval complete, 1 report results were downloaded.

@Kish26
Copy link
Author

Kish26 commented Sep 1, 2023

once report size is 1KB and payload is

{"status_code": 500, "headers": {}, "body": {"errors": [{"message": "'ascii' codec can't decode byte 0xc2 in position 1508426: ordinal not in range(128)", "code": 500}], "resources": []}}

@Kish26
Copy link
Author

Kish26 commented Sep 1, 2023

if it is configured to CSV, it is working but not with JSON. CSV is good for me. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jshcodes jshcodes

Labels
bug 🐛 Something isn't working code samples Functioning sample code to get you started reports Reporting questions or issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Version 1.3.1 - New Cloud Snapshots collection, FileVantage operations CrowdStrike/falconpy
3 participants
@Kish26 @jshcodes @cvjbrooks