Skip to content

Adds support for onebranch changes #233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gurkanindibay merged 7 commits into from
Jul 28, 2022
Merged

Adds support for onebranch changes #233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9cf8170
Adds procedure to be able to write postgres
gurkanindibay Jun 28, 2022
6bc4b76
Adds procedure to be able to validate warnings
gurkanindibay Jun 28, 2022
a37166b
Fixes dependency problems for bionic and centos
gurkanindibay Jul 19, 2022
014bd55
Adds container run enabled parameter
gurkanindibay Jul 25, 2022
b61e9f4
Fixes parameter addition problem
gurkanindibay Jul 26, 2022
a8ff3b8
Fixes statistic collector tests
gurkanindibay Jul 26, 2022
acf5f4b
Adds empty lines
gurkanindibay Jul 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion packaging_automation/citus_package.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,13 +232,14 @@ def build_package(github_token: non_empty(non_blank(str)),
docker_image_name = "packaging" if not is_test else "packaging-test"
postgres_extension = "all" if postgres_version == "all" else f"pg{postgres_version}"
os.environ["GITHUB_TOKEN"] = github_token
os.environ["CONTAINER_BUILD_RUN_ENABLED"] = "true"
Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This parameter is required to execute the packaging image in OneBranch.
Currently packaging images directly execute fetch_build_deb script and exits. However, OneBranch first needs to download and initialize then execute the pipeline scripts inside the container
This parameter is a switch to use the packaging images both in github actions and OneBranch pipelines
If this parameter is not set, so in OneBranch container initialisation phase, image does not exit just waits
If this is set, then the script executes as regular.

if not os.path.exists(input_output_parameters.output_dir):
os.makedirs(input_output_parameters.output_dir)

output = run_with_output(
f'docker run --rm -v {input_output_parameters.output_dir}:/packages -v '
f'{input_output_parameters.input_files_dir}:/buildfiles:ro -e '
f'GITHUB_TOKEN -e PACKAGE_ENCRYPTION_KEY -e UNENCRYPTED_PACKAGE '
f'GITHUB_TOKEN -e PACKAGE_ENCRYPTION_KEY -e UNENCRYPTED_PACKAGE -e CONTAINER_BUILD_RUN_ENABLED '
Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameter is added and set true to be able to execute build logic in fetch_and_build_deb/rpm scripts

f'citus/{docker_image_name}:{docker_platform}-{postgres_extension} {build_type.name}', text=True)

if output.stdout:
Expand Down
3 changes: 2 additions & 1 deletion packaging_automation/requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ GitPython
Jinja2
parameters_validation
pathlib2
psycopg2
psycopg2-binary
Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since out github actions main VM image is ubuntu focal, just majing this script work in ubuntu was enough
Now for OneBeanch, tools python requirements scripts shoule be executed in all supported images. In debian and centos images I got error so I needed to use binary version of psycopg2-binary

pycurl
PyGithub
pytest
Expand All @@ -16,3 +16,4 @@ urllib3
wheel
python-dotenv
prospector[with_everything]
setuptools==58
4 changes: 2 additions & 2 deletions packaging_automation/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# This file is autogenerated by pip-compile
# This file is autogenerated by pip-compile with python 3.8
Copy link
Copy Markdown
Member

@hanefi hanefi Jul 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, Consider introducing a .python-version file with the intended python version that should be used here.

Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#234

# To update, run:
#
# pip-compile
Expand Down Expand Up @@ -81,7 +81,7 @@ pluggy==0.13.1
# via pytest
prospector[with_everything]==1.5.0.1
# via -r requirements.in
psycopg2==2.9.1
psycopg2-binary==2.9.3
# via -r requirements.in
py==1.10.0
# via pytest
Expand Down
2 changes: 1 addition & 1 deletion packaging_automation/tests/test_package_cloud_statistics_collector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def get_filtered_package_count(session) -> int:
# Since package count for our test repo is lower than 100, we get the total package details by getting all the
# packages in one call
result = stat_get_request(
package_list_with_pagination_request_address(PACKAGE_CLOUD_PARAMETERS, 1, 100),
package_list_with_pagination_request_address(PACKAGE_CLOUD_PARAMETERS, 1, 200),
Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recently, I added new files into azure repository in packagecloud so this broke my tests. I increased the number for records in page to be able to get correct result

Copy link
Copy Markdown
Member

@hanefi hanefi Jul 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did we need to update this?

Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recently I put new packages into azure repo which exceeds 100 package . I'm using azure repo as my test repo to test the package counts. After exceed, my tests failed since some packages was outside the pagination limit.

RequestType.package_cloud_list_package, session)
package_info_list = json.loads(result.content)
package_list = list(filter(
Expand Down
15 changes: 15 additions & 0 deletions packaging_automation/validate_build_output.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import argparse
Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Github Actions, we validate package counts in citus_package.py without dependency for images. Since we can not execute citus_package, we needed to externalize validation logic for OneBranch build process. We call this function explicitly after fetch_and_build_deb/rpm script


from .packaging_warning_handler import (validate_output)
from .common_tool_methods import (PackageType)
from pathlib import Path

if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--output_file', required=True)
parser.add_argument('--ignore_file', required=True)
parser.add_argument('--package_type', choices=[p.name for p in PackageType], required=True)

args = parser.parse_args()
build_output = Path(args.output_file).read_text()
validate_output(build_output, args.ignore_file, PackageType[args.package_type])
11 changes: 11 additions & 0 deletions packaging_automation/write_postgres_versions_into_file.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import argparse
Copy link
Copy Markdown
Contributor Author

@gurkanindibay gurkanindibay Jul 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created because the same reason with validate_buildo_output. We execute this script just before fetch_and_build_deb/rpm to be able to get supported postgres versions


from .citus_package import (write_postgres_versions_into_file)

if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--project_version', required=True)
parser.add_argument('--input_files_dir', required=True)

args = parser.parse_args()
write_postgres_versions_into_file(args.input_files_dir,args.project_version)