Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

readme changes for recent makefile sample runs #45

Merged
merged 1 commit into from
May 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion transforms/code/code_quality/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ To run the samples, use the following `make` targets
* `run-cli-ray-sample` - runs src/code_quality_transform.py using command line args
* `run-local-ray-sample` - runs src/code_quality_local_ray.py
* `run-s3-ray-sample` - runs src/code_quality_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.
Expand Down
2 changes: 1 addition & 1 deletion transforms/code/malware/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ To run the samples, use the following `make` targets
* `run-local-sample` - runs src/malware_local.py
* `run-local-ray-sample` - runs src/malware_local_ray.py

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down
4 changes: 2 additions & 2 deletions transforms/code/proglang_select/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,9 @@ To run the samples, use the following `make` targets
* `run-local-sample` - runs src/proglang_select_local.py
* `run-local-ray-sample` - runs src/proglang_select_local_ray.py
* `run-s3-ray-sample` - runs src/proglang_select_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down
4 changes: 2 additions & 2 deletions transforms/universal/doc_id/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,9 @@ To run the samples, use the following `make` targets
* `run-local-sample` - runs src/doc_id_local.py
* `run-local-ray-sample` - runs src/doc_id_local_ray.py
* `run-s3-ray-sample` - runs src/doc_id_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down
4 changes: 2 additions & 2 deletions transforms/universal/ededup/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,9 @@ To run the samples, use the following `make` targets
* `run-cli-ray-sample` - runs src/ededup_transform.py using command line args
* `run-local-ray-sample` - runs src/ededup_local_ray.py
* `run-s3-ray-sample` - runs src/ededup_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down
4 changes: 2 additions & 2 deletions transforms/universal/fdedup/Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,9 +167,9 @@ To run the samples, use the following `make` targets
* `run-cli-ray-sample` - runs src/fdedup_transform.py using command line args
* `run-local-ray-sample` - runs src/fdedup_local_ray.py
* `run-s3-ray-sample` - runs src/fdedup_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down
4 changes: 2 additions & 2 deletions transforms/universal/filter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,9 +259,9 @@ To run the samples, use the following `make` targets
* `run-local-sample` - runs src/filter_local.py
* `run-local-ray-sample` - runs src/filter_local_ray.py
* `run-s3-ray-sample` - runs src/filter_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down
16 changes: 14 additions & 2 deletions transforms/universal/noop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,31 @@ The set of dictionary keys holding [NOOPTransform](src/noop_transform.py)
configuration for values are as follows:

* _noop_sleep_sec_ - specifies the number of seconds to sleep during table transformation.
* _noop_pwd_ - specifies a dummy password not included in metadata.

## Running

### Launched Command Line Options
When running the transform with the Ray launcher (i.e. TransformLauncher),
the following command line arguments are available in addition to
[the options provided by the launcher](../../../data-processing-lib/doc/launcher-options.md).
```
--noop_sleep_sec NOOP_SLEEP_SEC
Sleep actor for a number of seconds while processing the data frame, before writing the file to COS
--noop_pwd NOOP_PWD A dummy password which should be filtered out of the metadata
```
These correspond to the configuration keys described above.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-ray-sample` - runs src/noop_transform.py using command line args
* `run-local-sample` - runs src/noop_local.py
* `run-local-ray-sample` - runs src/noop_local_ray.py
* `run-s3-ray-sample` - runs src/noop_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down
4 changes: 2 additions & 2 deletions transforms/universal/tokenization/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ To run the samples, use the following `make` targets
* `run-cli-ray-sample` - runs src/tokenization_transform.py using command line args
* `run-local-ray-sample` - runs src/tokenization_local_ray.py
* `run-s3-ray-sample` - runs src/tokenization_s3_ray.py
* Requires prior invocation of `make minio-start minio-load` to load data into local minio for S3 access.
* Requires prior invocation of `make minio-start` to load data into local minio for S3 access.

These targes will activate the virtual environment and set up any configuration needed.
These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
Expand Down