Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update index.md #664

Merged
merged 2 commits into from
Feb 13, 2024
Merged

Update index.md #664

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0630834
Update index.md
abarciauskas-bgse Jan 10, 2024
c4892df
Update file_patterns.md
abarciauskas-bgse Jan 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/composition/file_patterns.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,8 +207,8 @@ pattern[index]
## From file pattern to `PCollection`

As covered in {doc}`index`, a recipe is composed of a sequence of Apache Beam transforms.
The data collection that Apache Beam transforms operates on is a
[`PCollection`](https://beam.apache.org/documentation/programming-guide/#pcollections).
The data Apache Beam transforms operate on are
[`PCollections`](https://beam.apache.org/documentation/programming-guide/#pcollections).
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So do the iterables of a filepattern become pcollections? This is a bit unclear to me.

Copy link
Member

@cisaacstern cisaacstern Jan 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this, @abarciauskas-bgse!

Yes, this is correct. A FilePattern is iterated over (via .items, not __iter__), and this is converted into a PCollection:

pattern = FilePattern(...)

recipe = beam.Create(pattern.items()) | ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@abarciauskas-bgse did you want to include any further detail as part of this PR? Or is the current change sufficient for now?

Therefore, we bring the contents of a `FilePattern` into a recipe, we pass the index:url
pairs generated by the file pattern's ``items()`` method into Beam's `Create` constructor
as follows:
Expand Down
2 changes: 1 addition & 1 deletion docs/composition/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

A recipe describes the steps to transform archival source data in one
format / location into analysis-ready, cloud-optimized (ARCO) data in another format /
location. Technically, a recipe is as a set of composite
location. Technically, a recipe is a composite of
[Apache Beam transforms](https://beam.apache.org/documentation/programming-guide/#transforms)
applied to the data collection associated with a {doc}`file pattern <file_patterns>`.
To write a recipe:
Expand Down