New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Processes the index projects in batches #612

Merged
merged 10 commits into from Sep 17, 2018

Continues processing rest YAML files encountered error processing one

   If there is an error processing any of the YAML file in index,
   log the error and continue processing rest of the YAML entries.

   Also updates the docstrings.
  • Loading branch information...
navidshaikh committed Sep 12, 2018
commit 480254d0b536dba30b328a0ac891260065f10e38
View
@@ -185,7 +185,8 @@ def read_yaml(self, filepath):
data = yaml.load(fin, Loader=yaml.BaseLoader)
except yaml.YAMLError as exc:
_print("Failed to read {}".format(filepath))
raise(exc)
_print("Error: {}".format(exc))
return None
else:

This comment has been minimized.

@bamachrn

bamachrn Sep 11, 2018

Collaborator

how are we handling this raise here we are raising exception from exception?

@bamachrn

bamachrn Sep 11, 2018

Collaborator

how are we handling this raise here we are raising exception from exception?

This comment has been minimized.

@navidshaikh

navidshaikh Sep 11, 2018

Collaborator

We are raising the exception but ensuring we print error message along with filepath with issue.

@navidshaikh

navidshaikh Sep 11, 2018

Collaborator

We are raising the exception but ensuring we print error message along with filepath with issue.

This comment has been minimized.

@navidshaikh

navidshaikh Sep 11, 2018

Collaborator

We have not handled this. Index Reader expects the yaml files to be in sane format. Do we want to log error and proceed further in this case as well?

@navidshaikh

navidshaikh Sep 11, 2018

Collaborator

We have not handled this. Index Reader expects the yaml files to be in sane format. Do we want to log error and proceed further in this case as well?

This comment has been minimized.

@bamachrn

bamachrn Sep 12, 2018

Collaborator

yeah, these should be taken care of by the index CI. But index reader should not fail entirely for single malformed entry. I would suggest we log it and continue to process other entries.

@bamachrn

bamachrn Sep 12, 2018

Collaborator

yeah, these should be taken care of by the index CI. But index reader should not fail entirely for single malformed entry. I would suggest we log it and continue to process other entries.

This comment has been minimized.

@navidshaikh

navidshaikh Sep 12, 2018

Collaborator

Agreed. I'll update the PR.

@navidshaikh

navidshaikh Sep 12, 2018

Collaborator

Agreed. I'll update the PR.

return data
@@ -202,6 +203,11 @@ def read_projects(self):
continue
app = self.read_yaml(yamlfile)
# if YAML file reading has failed, log the error and
# filename and continue processing rest of index
if not app:
continue
for entry in app['Projects']:
# create a project object here with all properties
try:
@@ -403,8 +409,19 @@ def list_builds_except(
List the builds except the phase(s) provided
default status=["Complete", "Failed"] <-- This will return
all the builds except the status.phase in ["Complete", "Failed"].
If provided a list of $filter_builds, it will filter mentioned builds
from outstanding builds. The builds name has build number string
appended, for eg seed-job-1, seed-job-2, thus filtering checks
if outstanding build name starts with given $filter_builds.
If status=[], return all the builds
:arg status: Status of outstanding builds to filter
:type status: List
:arg filter_builds: Builds to filter from outstanding builds
:type filter_builds: List
:return: List of outstanding builds
:rtype: List
"""
if not status:
return self.list_all_builds()
ProTip! Use n and p to navigate between commits in a pull request.