Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[backend/frontend] prevent delete CSV mappers if used by CSV ingesters (#6757) #6891

Merged
merged 2 commits into from
May 14, 2024

Conversation

labo-flg
Copy link
Member

@labo-flg labo-flg commented May 6, 2024

Proposed changes

  • when user deletes a CSV mapper, find and delete all csv ingesters that uses it.
  • add warning about feeds being deleted as part of the cascade deletion, in the confirm popover
  • fix bad form validation and reset in IngestionCsvCreation

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case (coverage and e2e)
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

@labo-flg labo-flg self-assigned this May 6, 2024
@labo-flg labo-flg added the filigran team use to identify PR from the Filigran team label May 6, 2024
Copy link

codecov bot commented May 6, 2024

Codecov Report

Attention: Patch coverage is 72.00000% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 67.41%. Comparing base (64d087e) to head (08c5fa5).

Files Patch % Lines
...rm/opencti-graphql/src/utils/migration-template.js 0.00% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6891      +/-   ##
==========================================
- Coverage   67.42%   67.41%   -0.01%     
==========================================
  Files         561      561              
  Lines       68510    68533      +23     
  Branches     5713     5713              
==========================================
+ Hits        46191    46200       +9     
- Misses      22319    22333      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@labo-flg labo-flg changed the title [backend/frontend] cascade delete CSV mappers into CSV ingesters (#6757) [backend/frontend] prevent delete CSV mappers if used by CSV ingesters (#6757) May 6, 2024
@labo-flg labo-flg force-pushed the issue/6757-csv-mappers branch 3 times, most recently from 3ea91d4 to ba03a89 Compare May 6, 2024 18:27
@labo-flg labo-flg marked this pull request as ready for review May 6, 2024 18:28
@labo-flg labo-flg marked this pull request as draft May 7, 2024 07:36
@labo-flg
Copy link
Member Author

labo-flg commented May 7, 2024

Back to draft: need to add a migration script to fix ingesters that might have lost their mapper.

@labo-flg labo-flg marked this pull request as ready for review May 7, 2024 15:49
@@ -657,7 +657,7 @@
"Do you want to delete this country?": "Souhaitez-vous supprimer ce pays ?",
"Do you want to delete this course of action?": "Souhaitez-vous supprimer cette conduite à suivre ?",
"Do you want to delete this CSV ingester?": "Voulez-vous supprimer cet ingéreur CSV ?",
"Do you want to delete this CSV mapper ?": "Voulez-vous supprimer ce mappage CSV ?",
"Do you want to delete this CSV mapper?": "Voulez-vous supprimer ce mappage CSV ?",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not related to the PR, but we would need at some point to review the auto translations. In french, CSV mapper has currently 3 different translations "mappage CSV", "mappeur CSV", "mapping CSV"

logApp.info(`${message} > started`);

const allIngesters = await listAllEntities(context, SYSTEM_USER, [ENTITY_TYPE_INGESTION_CSV]);
const mappersUsedIds = allIngesters.map((ingester) => ingester.csv_mapper_id);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor suggestion : maybe add a condition on allIngesters length before this line :
if (allIngesters.length > 0)
otherwise, no need to continue.

const ingesters = await listAllEntities<BasicStoreEntityIngestionCsv>(context, user, [ENTITY_TYPE_INGESTION_CSV], opts);
// prevent deletion if an ingester uses the mapper
if (ingesters.length > 0) {
throw FunctionalError('Cannot delete this CSV Mapper: it is used by one or more IngestionCsv ingester(s)', { id: csvMapperId });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe "CSV Feed" instead of "IngestionCsv" ? since it's the term we use in the platform

Copy link
Member

@SouadHadjiat SouadHadjiat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally ✅

  • migration works as expected, deleted csv feed without csv mapper
  • can't delete anymore csv mappers if used by a csv feed

@labo-flg labo-flg merged commit b547946 into master May 14, 2024
5 checks passed
@labo-flg labo-flg deleted the issue/6757-csv-mappers branch May 14, 2024 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filigran team use to identify PR from the Filigran team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CSV feeds screen broken after deleting the CSV mapper
2 participants