Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add enforce support for pipeline schedule #561

Merged
merged 11 commits into from Jul 15, 2023
23 changes: 21 additions & 2 deletions docs/reference/pipeline_schedules.md
Expand Up @@ -12,7 +12,13 @@ Additionally, under a `variables` key you can add the pipeline schedule variable

* Do not set `description` attribute - see [#535](https://github.com/gitlabform/gitlabform/issues/535#issue-1678509984)

Example:
There are 2 gitlabform specific keys/configs that can be set under `schedules` or individual schedule:

- `delete` - set this to `true` under a specific schedule to delete that particular schedule.
- `enforce` - set this to `true` under `schedules` so that any schedules that are not in `schedules` section are deleted.


Example 1:
```yaml
projects_and_groups:
group_1/project_1:
Expand All @@ -32,5 +38,18 @@ projects_and_groups:
other_variable:
value: another_value
"Obsolete schedule":
delete: true
delete: true # Delete this schedule
```

Example 2:
```yaml
projects_and_groups:
group_1/project_1:
schedules:
enforce: true # Delete all other pipeline schedules that exists for this project
"Some schedule":
ref: main
cron: "0 * * * MON-FRI"
cron_timezone: "London"
active: false
```
27 changes: 25 additions & 2 deletions gitlabform/processors/project/schedules_processor.py
Expand Up @@ -10,14 +10,22 @@
super().__init__("schedules", gitlab)

def _process_configuration(self, project_and_group: str, configuration: dict):
configured_schedules = configuration.get("schedules", {})
enforce_schedules = configuration.get("schedules|enforce", False)

Check warning on line 14 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L13-L14

Added lines #L13 - L14 were not covered by tests

# Remove 'enforce' key from the config so that it's not treated as a "schedule"
if enforce_schedules:
configured_schedules.pop("enforce")

Check warning on line 18 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L17-L18

Added lines #L17 - L18 were not covered by tests

existing_schedules = self.gitlab.get_all_pipeline_schedules(project_and_group)
schedule_ids_by_description = self.__group_schedule_ids_by_description(
existing_schedules
)

for schedule_description in sorted(configuration["schedules"]):
for schedule_description in sorted(configured_schedules):

Check warning on line 25 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L25

Added line #L25 was not covered by tests
schedule_ids = schedule_ids_by_description.get(schedule_description)
if configuration.get("schedules|" + schedule_description + "|delete"):

if configured_schedules[schedule_description].get("delete"):

Check warning on line 28 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L28

Added line #L28 was not covered by tests
if schedule_ids:
debug("Deleting pipeline schedules '%s'", schedule_description)
for schedule_id in schedule_ids:
Expand Down Expand Up @@ -66,6 +74,21 @@
configuration, project_and_group, schedule_description
)

if enforce_schedules:
debug("Delete unconfigured schedules because enforce is enabled")

Check warning on line 78 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L77-L78

Added lines #L77 - L78 were not covered by tests

for schedule in existing_schedules:
schedule_description = schedule["description"]
schedule_id = schedule["id"]

Check warning on line 82 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L80-L82

Added lines #L80 - L82 were not covered by tests

debug(f"processing {schedule_id}: {schedule_description}")
if schedule_description not in configured_schedules:
debug(

Check warning on line 86 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L84-L86

Added lines #L84 - L86 were not covered by tests
"Deleting pipeline schedule named '%s', because it is not in gitlabform configuration",
schedule_description,
)
self.gitlab.delete_pipeline_schedule(project_and_group, schedule_id)

Check warning on line 90 in gitlabform/processors/project/schedules_processor.py

View check run for this annotation

Codecov / codecov/patch

gitlabform/processors/project/schedules_processor.py#L90

Added line #L90 was not covered by tests

def create_schedule_with_variables(
self, configuration, project_and_group, schedule_description
):
Expand Down
89 changes: 89 additions & 0 deletions tests/acceptance/standard/test_schedules.py
Expand Up @@ -199,6 +199,95 @@ def test__delete_schedule(self, project, schedules):
)
assert schedule is None

def test__schedule_enforce_new_schedule(self, project, schedules):
new_schedule = f"""
projects_and_groups:
{project.path_with_namespace}:
schedules:
enforce: true
"New schedule to test enforce config":
ref: main
cron: "30 1 * * *"
"""

schedules_before = project.pipelineschedules.list()

run_gitlabform(new_schedule, project)

schedules_after = project.pipelineschedules.list()

schedule = self.__find_pipeline_schedule_by_description_and_get_first(
project, "New schedule to test enforce config"
)
assert len(schedules_before) == 6
assert len(schedules_after) == 1
assert schedule is not None
assert schedule.description == "New schedule to test enforce config"
assert schedule.ref == "main"
assert schedule.cron == "30 1 * * *"
assert schedule.cron_timezone == "UTC"
assert schedule.active is True

def test__schedule_enforce_new_and_existing_schedule(self, project, schedules):
new_schedule = f"""
projects_and_groups:
{project.path_with_namespace}:
schedules:
enforce: true
"New schedule to test enforce config":
ref: main
cron: "30 1 * * *"
"New schedule with variables":
ref: main
cron: "30 1 * * *"
variables:
var1:
value: value123
var2:
value: value987
variable_type: file
"""

schedules_before = project.pipelineschedules.list()

run_gitlabform(new_schedule, project)

schedules_after = project.pipelineschedules.list()

schedule1 = self.__find_pipeline_schedule_by_description_and_get_first(
project, "New schedule to test enforce config"
)
schedule2 = self.__find_pipeline_schedule_by_description_and_get_first(
project, "New schedule with variables"
)
assert len(schedules_before) == 1
assert len(schedules_after) == 2
Comment on lines +263 to +264
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gdubicki - It seems the tests are currently setup to be dependent on earlier test case's results. That's why this assertion looks odd to me. I expected each test case to be self-contained and not affecting other tests. I'm not sure if this was done intentionally. Thought I'd mention.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a result of the fact that we are often using fixtures that have a pytest scope higher than "function". Particularly, the fixture for a project has scope "class", so the schedules created in one method of a test class such as TestSchedules will in fact be retained in the next test method.

To fix that we would have to use "function" scope for an almost perfect separation of each test but we are are talking about tests on a real-world GitLab instance where creating and deleting projects in GitLab is pretty slow.

I just did a test of how this would work out in practice for test_schedules.py:

Code as-is, with project and group having scope "class":

1st run - 7.708s
2nd run - 6.280s
3rd run - 7.390s

Code with project and group scope changed to "function":

1st run - 14.556s
2nd run - 18.286s
3rd run - 18.335s

So the tests time effectively increases 2-2.5 times.

The total current time of running tests in GitHub Actions is ~3 minutes for standard and ~1.5 minutes for premium ones, so including the ~4 minutes of GitLab provisioning time the whole pipeline time would increase from ~7 minutes to ~10 minutes...

Because of that we are now using 2 ways of dealing with the issue of tests affecting each other, only where it affects the tests:

a) use "function" scoped fixtures on specific single tests - see the uses of project_for_function fixture,
b) make the assertions about the initial state more relaxed - see https://github.com/gitlabform/gitlabform/blob/main/tests/acceptance/standard/test_members.py#L12-L12 as an example.

Of course this is not perfect, but it's the best compromise between correctness and performance of the tests that I was able to develop at the time. If you think that we should make changes then feel free to share your ideas, @amimas. :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gdubicki for the details and the performance tests. I also realized this came down to fixture scope after looking into it a bit, but didn't have time to learn what changes are needed if the scope is changed.

I'm starting to think what you've got right now actually makes sense. These are not mocked unit tests. As you said, we're running these against a real GitLab instance. Also not sure if there's a real need for running only individual test as opposed to running all tests in a single file. That would probably make the testing and development faster, but is it worth it? So, it is a balance between isolated tests vs performance of entire test suites.

I don't have any other ideas at the moment. Being new to the code base, it was a bit confusing at the beginning. I had to go through existing test cases and then figure out if the assertions in the new test case makes sense. Maybe what might help others is if we have additional sections in the contributing docs about general setup/expectations of acceptance tests. I have only played with 1 test file through this PR so far. If I can think of any improvements to the docs/development guide in future, I'll open new PR accordingly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, we should at least document it. I'll try to do it soon.


assert schedule1 is not None
assert schedule1.description == "New schedule to test enforce config"
assert schedule1.ref == "main"
assert schedule1.cron == "30 1 * * *"
assert schedule1.cron_timezone == "UTC"
assert schedule1.active is True

assert schedule2 is not None
assert schedule2.description == "New schedule with variables"
assert schedule2.ref == "main"
assert schedule2.cron == "30 1 * * *"
assert schedule2.cron_timezone == "UTC"
assert schedule2.active is True

variables = schedule2.attributes["variables"]
assert variables is not None
assert len(variables) == 2
assert variables[0]["variable_type"] == "env_var"
assert variables[0]["key"] == "var1"
assert variables[0]["value"] == "value123"

assert variables[1]["variable_type"] == "file"
assert variables[1]["key"] == "var2"
assert variables[1]["value"] == "value987"

@classmethod
def __find_pipeline_schedule_by_description_and_get_first(
cls, project, description
Expand Down