Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-13786] [Playground] [Bugfix] Update CI/CD to verify only single-file examples #16701

Merged
merged 3 commits into from
Feb 7, 2022
Merged

[BEAM-13786] [Playground] [Bugfix] Update CI/CD to verify only single-file examples #16701

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6abc731
Update CI/CD to verify only single-file examples
pavel-avilov Feb 2, 2022
d224464
Refactoring code
pavel-avilov Feb 2, 2022
4f8e1af
list comprehension to filter
pavel-avilov Feb 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions playground/infrastructure/cd_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,18 @@ class CDHelper:

It is used to save beam examples/katas/tests and their output on the GCS.
"""

def store_examples(self, examples: List[Example]):
"""
Store beam examples and their output in the Google Cloud.

Outputs for multifile examples are left empty.
"""
logging.info("Start of executing Playground examples ...")
asyncio.run(self._get_outputs(examples))
logging.info("Finish of executing Playground examples")
single_file_examples = [
example for example in examples if example.tag.multifile is False
]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about use filter function ro filtering?

Suggested change
single_file_examples = [
example for example in examples if example.tag.multifile is False
]
single_file_examples = filter(lambda example: example.multifile, examples)

Just remember, that filter returns a filter object, that can be iterated, but is not available for slices

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

logging.info("Start of executing only single-file Playground examples ...")
asyncio.run(self._get_outputs(single_file_examples))
logging.info("Finish of executing single-file Playground examples")

logging.info("Start of sending Playground examples to the bucket ...")
self._save_to_cloud_storage(examples)
Expand All @@ -74,12 +78,13 @@ async def _get_outputs(self, examples: List[Example]):
tasks = [client.get_log(example.pipeline_id) for example in examples]
logs = await asyncio.gather(*tasks)

if len(examples) > 0 and (examples[0].sdk is SDK_PYTHON or examples[0].sdk is SDK_JAVA):
if len(examples) > 0 and (examples[0].sdk is SDK_PYTHON or
examples[0].sdk is SDK_JAVA):
tasks = [client.get_graph(example.pipeline_id) for example in examples]
graphs = await asyncio.gather(*tasks)

for graph, example in zip(graphs, examples):
example.graph = graph
example.graph = graph

for output, example in zip(outputs, examples):
example.output = output
Expand Down
7 changes: 4 additions & 3 deletions playground/infrastructure/ci_cd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@

def _ci_step(examples: List[Example]):
"""
CI step to verify all beam examples/tests/katas
CI step to verify single-file beam examples/tests/katas
"""

ci_helper = CIHelper()
Expand Down Expand Up @@ -84,9 +84,10 @@ def _run_ci_cd(step: config.Config.CI_CD_LITERAL, sdk: Sdk):
logging.info("Number of found Playground examples: %s", len(examples))

if step == config.Config.CI_STEP_NAME:
logging.info("Start of verification Playground examples ...")
logging.info(
"Start of verification only single_file Playground examples ...")
_ci_step(examples=examples)
logging.info("Finish of verification Playground examples")
logging.info("Finish of verification single_file Playground examples")
if step == config.Config.CD_STEP_NAME:
logging.info("Start of storing Playground examples ...")
_cd_step(examples=examples)
Expand Down
8 changes: 5 additions & 3 deletions playground/infrastructure/ci_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,12 @@ async def verify_examples(self, examples: List[Example]):

1. Find all beam examples starting from directory os.getenv("BEAM_ROOT_DIR")
2. Group code of examples by their SDK.
3. Run processing for all examples to verify examples' code.
3. Run processing for single-file examples to verify examples' code.
"""
await get_statuses(examples)
await self._verify_examples(examples)
single_file_examples = [example for example in examples
if example.tag.multifile is False]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same, can we use the filter for filtering? :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

await get_statuses(single_file_examples)
await self._verify_examples(single_file_examples)

async def _verify_examples(self, examples: List[Example]):
"""
Expand Down