Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attempting to delete a missing report causes server crash #171363

Closed
tsullivan opened this issue Nov 15, 2023 · 2 comments · Fixed by #171858
Closed

Attempting to delete a missing report causes server crash #171363

tsullivan opened this issue Nov 15, 2023 · 2 comments · Fixed by #171858
Assignees
@vadimkibana
Labels
bug Fixes for quality problems that affect the customer experience Feature:Reporting Reporting (PDF, CSV, ..) feature Team:SharedUX Team label for AppEx-SharedUX (formerly Global Experience)

Comments

@tsullivan
Copy link
Member

Kibana version: 8.12.0-SNAPSHOT

Elasticsearch version:

Server OS version: MacOS/Arm

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.): source

Description of the problem including expected versus actual behavior:

Steps to reproduce:

  1. Generate some reports
  2. Navigate to Stack Management
  3. Select the reports in the listing
  4. Click the Delete button
  5. Before the screen has refreshed, click the Delete button again
  6. This could cause the server to crash

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[2023-11-15T14:54:36.697-07:00][DEBUG][plugins.reporting.runTask] Reports running: 0.
[2023-11-15T14:54:56.058-07:00][DEBUG][plugins.reporting.content_stream.looonh0d0xilc137af7n1l40] Chunk size is 78642432 bytes.
[2023-11-15T14:54:56.065-07:00][DEBUG][plugins.reporting.content_stream.looonh0d0xilc137af7n1l40] Updating report contents.
[2023-11-15T14:55:03.281-07:00][DEBUG][plugins.reporting.content_stream.looonh0d0xilc137af7n1l40] Chunk size is 78642432 bytes.
[2023-11-15T14:55:03.286-07:00][DEBUG][plugins.reporting.content_stream.looonh0d0xilc137af7n1l40] Updating report contents.
[2023-11-15T14:55:03.314-07:00][ERROR][plugins.reporting] ResponseError: document_missing_exception
        Root causes:
                document_missing_exception: [looonh0d0xilc137af7n1l40]: document missing
    at KibanaTransport.request (/Users/tim/elastic/kibana/node_modules/@elastic/transport/src/Transport.ts:535:17)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
{
  name: 'ResponseError',
  message: 'document_missing_exception\n' +
    '\tRoot causes:\n' +
    '\t\tdocument_missing_exception: [looonh0d0xilc137af7n1l40]: document missing'
}
 server crashed  with status code 1
image
@botelastic botelastic bot added the needs-team Issues missing a team label label Nov 15, 2023
@tsullivan tsullivan added bug Fixes for quality problems that affect the customer experience Feature:Reporting Reporting (PDF, CSV, ..) feature Team:SharedUX Team label for AppEx-SharedUX (formerly Global Experience) labels Nov 15, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/appex-sharedux (Team:SharedUX)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Nov 15, 2023
@vadimkibana
Copy link
Contributor

vadimkibana commented Nov 20, 2023 •
edited

From team sync:

Easier to reproduce on serverless.

@vadimkibana vadimkibana self-assigned this Nov 20, 2023
@vadimkibana vadimkibana mentioned this issue Nov 23, 2023
Merged
dej611 pushed a commit to dej611/kibana that referenced this issue Nov 28, 2023
@vadimkibana @kibanamachine
[Reporting] Prevent server from crashing due to concurrent deletes (e…
df5383e 
…lastic#171858)

## Summary

Closes elastic#171363

- The thing that crashes the server is `refresh: 'wait_for'`.
- I've also changed `await promisify` in one place, as that looked risky
to me.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@tsullivan tsullivan mentioned this issue Dec 20, 2023
Merged
1 task
tsullivan added a commit that referenced this issue Jan 2, 2024
@tsullivan
Handle content stream errors in report pre-deletion (#173792)
dc813c3 
Re-addresses #171363

The bug was still evident, especially when using network throttling to
add slight lag to the request turnaround times.

This PR adds more handling of errors that could be thrown slightly prior
to deleting the report document, when we try to clear all chunks of the
report using the content stream.

<details>
<summary>Before</summary>



https://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f


</details>

<details>
<summary>After</summary>


https://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78


</details>

### Checklist
- [x] Unit tests
tsullivan added a commit to tsullivan/kibana that referenced this issue Jan 2, 2024
@tsullivan
Handle content stream errors in report pre-deletion (elastic#173792)
bf70e08 
Re-addresses elastic#171363

The bug was still evident, especially when using network throttling to
add slight lag to the request turnaround times.

This PR adds more handling of errors that could be thrown slightly prior
to deleting the report document, when we try to clear all chunks of the
report using the content stream.

<details>
<summary>Before</summary>

https://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f

</details>

<details>
<summary>After</summary>

https://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78

</details>

### Checklist
- [x] Unit tests

(cherry picked from commit dc813c3)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jan 2, 2024
@tsullivan
Handle content stream errors in report pre-deletion (elastic#173792)
e03d70b 
Re-addresses elastic#171363

The bug was still evident, especially when using network throttling to
add slight lag to the request turnaround times.

This PR adds more handling of errors that could be thrown slightly prior
to deleting the report document, when we try to clear all chunks of the
report using the content stream.

<details>
<summary>Before</summary>

https://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f

</details>

<details>
<summary>After</summary>

https://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78

</details>

### Checklist
- [x] Unit tests

(cherry picked from commit dc813c3)
tsullivan added a commit to tsullivan/kibana that referenced this issue Jan 2, 2024
@tsullivan
Handle content stream errors in report pre-deletion (elastic#173792)
41fd648 
Re-addresses elastic#171363

The bug was still evident, especially when using network throttling to
add slight lag to the request turnaround times.

This PR adds more handling of errors that could be thrown slightly prior
to deleting the report document, when we try to clear all chunks of the
report using the content stream.

<details>
<summary>Before</summary>

https://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f

</details>

<details>
<summary>After</summary>

https://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78

</details>

### Checklist
- [x] Unit tests

(cherry picked from commit dc813c3)

# Conflicts:
#	x-pack/plugins/reporting/server/routes/common/jobs/get_job_routes.ts
kibanamachine added a commit that referenced this issue Jan 3, 2024
@kibanamachine @tsullivan
[8.12] Handle content stream errors in report pre-deletion (#173792) (#…
73951f9 
…174139)

# Backport

This will backport the following commits from `main` to `8.12`:
- [Handle content stream errors in report pre-deletion
(#173792)](#173792)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Tim
Sullivan","email":"tsullivan@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-01-02T23:00:53Z","message":"Handle
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","backport:prev-MAJOR","v8.13.0"],"title":"Handle
content stream errors in report
pre-deletion","number":173792,"url":"#173792
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"#173792
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c"}}]}] BACKPORT-->

Co-authored-by: Tim Sullivan <tsullivan@users.noreply.github.com>
tsullivan added a commit that referenced this issue Jan 4, 2024
@tsullivan
[8.11] Handle content stream errors in report pre-deletion (#173792) (#…
f5436fc 
…174140)

# Backport

This will backport the following commits from `main` to `8.11`:
- [Handle content stream errors in report pre-deletion
(#173792)](#173792)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Tim
Sullivan","email":"tsullivan@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-01-02T23:00:53Z","message":"Handle
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","backport:prev-MAJOR","v8.12.0","v8.13.0","v8.11.4"],"number":173792,"url":"#173792
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c"}},"sourceBranch":"main","suggestedTargetBranches":["8.12","8.11"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"#173792
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c"}},{"branch":"8.11","label":"v8.11.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
tsullivan added a commit that referenced this issue Jan 4, 2024
@tsullivan
[7.17] Handle content stream errors in report pre-deletion (#173792) (#…
eb687c4 
…174141)

# Backport

This will backport the following commits from `main` to `7.17`:
- [Handle content stream errors in report pre-deletion
(#173792)](#173792)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Tim
Sullivan","email":"tsullivan@users.noreply.github.com"},"sourceCommit":{"committedDate":"2024-01-02T23:00:53Z","message":"Handle
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:prev-minor","backport:prev-MAJOR","v8.12.0","v8.13.0","v8.11.4"],"number":173792,"url":"#173792
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c"}},"sourceBranch":"main","suggestedTargetBranches":["8.12","8.11"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"#173792
content stream errors in report pre-deletion (#173792)\n\nRe-addresses
#171363 bug was still
evident, especially when using network throttling to\r\nadd slight lag
to the request turnaround times.\r\n\r\nThis PR adds more handling of
errors that could be thrown slightly prior\r\nto deleting the report
document, when we try to clear all chunks of the\r\nreport using the
content
stream.\r\n\r\n<details>\r\n<summary>Before</summary>\r\n\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/c27fe314-0f93-42b4-8076-99a1e30b8d2f\r\n\r\n\r\n</details>\r\n\r\n<details>\r\n<summary>After</summary>\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/908371/4c1f5edd-73f1-4ca4-a40a-f900ca5f9c78\r\n\r\n\r\n</details>\r\n\r\n###
Checklist\r\n- [x] Unit
tests","sha":"dc813c351fe111c895e85a188372ad31625d8c8c"}},{"branch":"8.11","label":"v8.11.4","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vadimkibana vadimkibana

Labels
bug Fixes for quality problems that affect the customer experience Feature:Reporting Reporting (PDF, CSV, ..) feature Team:SharedUX Team label for AppEx-SharedUX (formerly Global Experience)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Reporting] Prevent server from crashing due to concurrent deletes vadimkibana/kibana
3 participants
@tsullivan @elasticmachine @vadimkibana