Job tasks' stuck in "Scheduled"/"Waiting" state

[julianjimenez-ms]

Information

• Onefuzz version: 5.7.1
• OS: Windows 11

Provide detailed reproduction steps (if any)

1. Deploy OneFuzz

PS> python3.9.exe .\deploy.py eastus2 <project>-onefuzz OneFuzz-<project> <email> .\config.json
PS> onefuzz config --endpoint <...> --authority <...> --client_id <...>
PS> onefuzz pools create <project>-onefuzz-pool windows --query pool_id
PS> onefuzz scalesets create <project>-onefuzz-pool 1
{
  "ephemeral_os_disks": false,
  "image": "MicrosoftWindowsDesktop:Windows-10:20h2-pro:latest",
  "needs_config_update": false,
  "pool_name": "<project>-onefuzz-pool",
  "region": "eastus2",
  "scaleset_id": "<...>",
  "size": 1,
  "spot_instances": false,
  "state": "init",
  "tags": {},
  "vm_sku": "Standard_D2s_v3"
}

2. Launch a libfuzzer job with the basic template

PS> onefuzz template libfuzzer basic <project> <test-name> build-1 <project>-onefuzz-pool --target_exe <fuzz-target>.exe
INFO:onefuzz:creating libfuzzer from template
INFO:onefuzz:creating job (runtime: 24 hours)
INFO:onefuzz:created job: <job-id>
INFO:onefuzz:using container: oft-setup-<...>
INFO:onefuzz:using container: oft-inputs-<...>
INFO:onefuzz:using container: oft-crashes-<...>
INFO:onefuzz:using container: oft-reports-<...>
INFO:onefuzz:using container: oft-unique-reports-<...>
INFO:onefuzz:using container: oft-unique-inputs-<...>
INFO:onefuzz:using container: oft-no-repro-<...>
INFO:onefuzz:using container: oft-coverage-<...>
INFO:onefuzz:using container: oft-regression-reports-<...>
INFO:onefuzz:uploading target exe <fuzz-target>.exe
INFO:onefuzz:creating libfuzzer_regression task
INFO:onefuzz:creating libfuzzer task
INFO:onefuzz:creating coverage task
INFO:onefuzz:creating libfuzzer_crash_report task
INFO:onefuzz:done creating tasks
{
  "config": {
    "build": "build-1",
    "duration": 24,
    "logs": "<...>",
    "name": "<test-name>",
    "project": "<project>"
  },
  "job_id": "<...>",
  "state": "init",
  "user_info": {
    "application_id": "<...>",
    "object_id": "<...>",
    "upn": "<email>"
  }
}

3. Check the status of the tasks in this job (>20h after launching it)

PS> onefuzz jobs get <job-id>
{
    "config": {
        "build": "build-1",
        "duration": 24,
        "logs": "<...>",
        "name": "<test-name>",
        "project": "<project>"
    },
    "job_id": "<job-id>",
    "state": "enabled",
    "task_info": [
        {
            "state": "scheduled",
            "task_id": "<...>",
            "type": "libfuzzer_fuzz"
        },
        {
            "state": "scheduled",
            "task_id": "<...>",
            "type": "libfuzzer_regression"
        },
        {
            "state": "waiting",
            "task_id": "<...>",
            "type": "libfuzzer_crash_report"
        },
        {
            "state": "waiting",
            "task_id": "<...>",
            "type": "coverage"
        }
    ],
    "user_info": {
        "application_id": "<...>",
        "object_id": "<...>",
        "upn": "<email>"
    }
}

Expected result

The tasks start after a few seconds or minutes, given that the VM is provisioned and running with no apparent issues. I might be missing something; in which case I think that the Getting Started docs are lacking an explanation.

Actual result

The tasks haven't started even after more than 20h and I haven't seen any error message indicating a problem.

AB#40775714

[mgreisen]

From Cheick:

Try increasing the size of your scaleset to at least 4. The default template requires 2 VM's for LibFuzzer, 1 for coverage, 1 for crash report. You can query the task configuration to get those details with onefuzz task get <task_id>

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[julianjimenez-ms]

Hi,

Thanks for following up but unfortunately this didn't work.

I've re-created the scaleset with 4 VMs and re-launched the job. There was a single VM instance in the Azure portal and another 3 have been created shortly after. All show up as provisioned successfully, even though according to the table the first one is not the latest model and the other 3 are.

The tasks in the new job are again all stuck in scheduled/waiting status after a couple of hours.

How can I gather more information?

[chkeita]

You can check the in Transaction Search of the AppInsight instance of the deployment to see if there are any issues related to the job.
Use the job_id or the scaleset_id as a filter

[julianjimenez-ms]

I don't see any issues. These are the last two traces:

tasks not ready: 2

Executed 'Functions.timer_tasks' (Succeeded, Id=<...>, Duration=493ms)

There are also some of these before them

sending event: EventType.task_state_updated - {
  "job_id": "<...>",
  "task_id": "<...>",
  "state": "scheduled",
  "config": {
    "job_id": "<...>",
    "task": {
      "type": "libfuzzer_regression",
      "duration": 24,
      "target_exe": "<...>FuzzTarget.exe",
      "target_options": [],
      "target_options_merge": false,
      "check_asan_log": false,
      "check_debugger": true,
      "check_fuzzer_help": true,
      "rename_output": false,
      "reboot_after_setup": false,
      "preserve_existing_outputs": false
    },
    "pool": {
      "count": 1,
      "pool_name": "<...>-onefuzz-pool"
    },
    "containers": [{
      "type": "setup",
      "name": "oft-setup-6dde58eda9395cb09c34138d17145b6e"
    }, {
      "type": "crashes",
      "name": "oft-crashes-7480911c0aed564f890a065599b25fb0"
    }, {
      "type": "unique_reports",
      "name": "oft-unique-reports-7480911c0aed564f890a065599b25fb0"
    }, {
      "type": "regression_reports",
      "name": "oft-regression-reports-9f84629d64fe5905a7d16f053d797b93"
    }],
    "tags": {
      "project": "<...>",
      "name": "<...>",
      "build": "build-1"
    },
    "colocate": true
  }
}

[julianjimenez-ms]

@chkeita Any further insights on what might be happening here?