Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #21321: Add API endpoint to import an archive of rules/etc #4367

Conversation

fanf
Copy link
Member

@fanf fanf commented Jul 7, 2022

https://issues.rudder.io/issues/21247
(before: https://issues.rudder.io/issues/21321)

This PR add the possiblity to export and import via upload zip archives to rudder.

Limitation are:

  • we don't check much for now. Things that should be tested: technique and version used in directive, existence of provided group id for new groups, existence of groups/dierctives used in rule.
  • the save is not even a bit atomic. We save technique, reload technique library, then directives, then groups, then rules. All that sould be done in one go, but we don't have underlying infrastructure yet. It might prove to be a showstopper that needs to be addressed before release (because here, we may start semi generation). An alternative would be to write everything in git, then "reload from master", but I have doubts making that the default behavior for archive upload, without someone to take care of the mess if something goes wrong (concurrent change for ex)

Main arch change:

  • created an EventMetadata to factor out changeId, actor, option msg. We should use it everywhere in the future, it's so much lighter. Perhaps find a shorter name
  • some change in ZioUtils#accumulate to remove some dependencies on cats (I was needed some variation of what we had already)
  • move the low level part of writting tehcniques (ie the commit part) in git archiver file, change a bit its signature
  • update a lot of parts of mocking framework to make it usable with zip upload - also handle some cases were binary content were causing unhandled errors in our API

For the export part in that PR

The syntax for the API is:

 Request format:
   ../archives/export/rules=rule_ids&directives=dir_ids&techniques=tech_ids&groups=group_ids&include=scope
 Where:
 - rule_ids = xxxx-xxxx-xxx-xxx[,other ids]
 - dir_ids = xxxx-xxxx-xxx-xxx[,other ids]
 - group_ids = xxxx-xxxx-xxx-xxx[,other ids]
 - tech_ids = techniqueName/1.0[,other tech ids]
- scope = all (default), none, directives, techniques (implies directive), groups

Ssytem items are filtered out, it seems that it would cause more pain an complication than the opposite (with in addition the case of special target to handle).

Interesting parts:

  • zip is an horrible format.
  • there is a feature switch to enable the feature in settings (rudder_featureSwitch_archiveApi=enalbed,disabled)
  • we use item names and not UUID for file names (UUID is still of course in the serialized json), with a simple name mangling that replace non ascii chars to their ascii correspondance (as utf and java intended)
  • revision works, you can ask for specific revision of items (but the + in url must be encoded as %2B because urls)

@fanf fanf requested a review from VinceMacBuche July 7, 2022 15:40
@fanf
Copy link
Member Author

fanf commented Jul 7, 2022

PR updated with a new commit

@VinceMacBuche
Copy link
Member

I think it's ok to me apart the comment with url format with a typo (/ instead of ? ) Do you want the documentation of the api to be done in another PR ?

@fanf
Copy link
Member Author

fanf commented Jul 8, 2022

PR rebased

@fanf fanf force-pushed the ust_21321/add_api_endpoint_to_import_an_archive_of_rules_etc branch from 5f53f70 to f30a1a4 Compare July 8, 2022 15:41
@fanf
Copy link
Member Author

fanf commented Jul 8, 2022

It seems that I lose the documentation at some point, I think I forgot to add the changes :/
It was rather long to do, I will do it in an other pr with ticket: https://issues.rudder.io/issues/21394

@Normation-Quality-Assistant
Copy link
Contributor

OK, merging this PR

@Normation-Quality-Assistant Normation-Quality-Assistant merged commit f30a1a4 into Normation:master Jul 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants