Add text/diff/data ops, YAML/Parquet, OTel tracing, and OneDrive/Box backends#59
Merged
Add text/diff/data ops, YAML/Parquet, OTel tracing, and OneDrive/Box backends#59
Conversation
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 383 |
| Duplication | 14 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Batch A of the new-ops initiative — stdlib-only, no new dependencies. - text_ops: FA_file_split, FA_file_merge, FA_encoding_convert, FA_line_count, FA_sed_replace (literal + regex). All writes atomic. - diff_ops: FA_diff_files / FA_diff_dirs / FA_apply_patch. Extends the existing diff_text_files / diff_dirs helpers with apply_text_patch (unified-diff driver with strict context-line verification) and a JSON-friendly diff_dirs_summary wrapper. - data_ops: FA_csv_filter (column projection + where clause), FA_csv_to_jsonl, FA_jsonl_iter (with limit), FA_jsonl_append. - Facade + __all__ updated; new exception types TextOpsException and DataOpsException added under the shared exceptions hierarchy. Tests: 42 new cases — all pass. Full suite: 686 passed, 8 skipped.
Batch B — introduces three required dependencies: PyYAML, pyarrow, opentelemetry-api/sdk (all added to requirements.txt + dev.toml + stable.toml). YAML / Parquet: - FA_yaml_get, FA_yaml_set, FA_yaml_delete — dotted-path access with yaml.safe_load (never yaml.load) so untrusted config can't construct arbitrary Python objects. - FA_parquet_read (with limit + columns projection), FA_parquet_write, FA_csv_to_parquet. All writes atomic. Tracing: - core.tracing.init_tracing installs a global TracerProvider with a configurable SpanExporter (defaults to a null exporter so spans can still be inspected without wiring up a backend). - action_span context manager is a zero-cost no-op until tracing is initialised. - ActionExecutor._execute_event wraps every action in an "automation_file.action" span tagged with fa.action=<name>. - FA_tracing_init exposed through the registry. Tests: 22 new — all pass. Full suite: 708 passed, 8 skipped.
Batch C — two new cloud backends mirroring the existing S3 / Azure / Dropbox pattern, plus matching PySide6 tabs. OneDrive (Microsoft Graph via MSAL): - OneDriveClient holds an OAuth2 access token and a requests.Session, exposes both later_init(token) and device_code_login(client_id) paths. graph_request() centralises auth header + error wrapping. - Ops: upload_file / upload_dir (4 MiB simple-upload cap), download_file, delete_item, list_folder (follows @odata.nextLink). - FA_onedrive_* actions registered, UI tab added to Transfer sidebar and Home status probes. Box (boxsdk): - BoxClient wraps boxsdk.OAuth2 + Client with access-token init. - Ops: upload_file (returns new file id), upload_dir (flattens tree since Box requires pre-existing folder ids), download_file, delete_file, delete_folder (recursive flag), list_folder. - FA_box_* actions registered, UI tab added. Deps added to requirements.txt + stable.toml + dev.toml: msal, boxsdk. Both declared as required runtime deps, consistent with the existing boto3 / azure-storage-blob / dropbox / paramiko treatment. Tests: 31 new — all pass. Full suite: 739 passed, 8 skipped. Real Graph / Box endpoints are out of CI; tests exercise registry wiring, guard clauses, and exception-wrapping paths via in-memory fakes.
JE-Chen
force-pushed
the
feat/new-ops-and-backends
branch
from
81737e5 to
8995da1
Compare
CI lint (mypy, 9 errors): - text_ops.file_merge: replace the iter(lambda) chunker with a plain while loop so mypy can infer the chunk type. - data_ops: _resolve_fieldnames now accepts Sequence[str] | None to match csv.DictReader.fieldnames. - tracing: drop unused # type: ignore[attr-defined] comments now that mypy resolves the Once / TracerProvider references on its own. - tracing: replace # type: ignore[override] on the _NullExporter methods with plain signatures that already match the parent. - box.upload_ops: wrap box_upload_file in a helper that returns bool so walk_and_upload's Callable[[Path, str], bool] is satisfied. Codacy (11 new issues): - Bandit B105 in test_onedrive_ops: promote the literal "fake-token" string to a named local and mark with # nosec B105 + comment. - PyLint W0212 across test_onedrive_ops + test_tracing: add file-level # pylint: disable=protected-access with rationale — the tests legitimately poke tracing._state and OneDriveClient._session / _access_token to inject fakes. - PyLint R0914 in data_ops.csv_filter (17 locals): refactor into csv_filter → _stream_csv_filter → _write_filtered_rows, each below the 15-locals cap. - PyLint R0913 in onedrive.client.graph_request (8 args): collapse params / json_body / data / headers into **request_kwargs forwarded to requests.Session.request. - PyLint R0917 too-many-positional-arguments on the new data_ops helpers: justified disable-next per helper. SonarCloud (4 issues): - python:S108 empty-body blocks in test_tracing: replace the `pass`-only with blocks with meaningful assertions / operations so the span bodies are non-empty. - pythonsecurity:S2083 in apply_text_patch: document that the caller is the trust boundary for target (matching the rest of the local ops) and mark the write_text call with # NOSONAR pythonsecurity:S2083 + explanation. - python:S3776 cognitive complexity in _apply_unified_patch (30) and _iter_hunks (17): split both into small helpers (_copy_up_to, _apply_hunk_ops, _verify_live_line for the former; a _HunkParseState dataclass + _consume_patch_line for the latter). Each helper now scores well under the 15-complexity cap. Verified: `ruff check`, `ruff format --check`, `mypy`, and `pytest` (739 passed, 8 skipped).
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three-batch feature push covering new local ops, structured-data support, tracing, and two additional cloud backends. 105 → 137 registered actions (+32). Full test suite: 739 passed, 8 skipped.
Batch A — stdlib-only ops (commit c982a46)
FA_file_split,FA_file_merge,FA_encoding_convert,FA_line_count,FA_sed_replace(literal + regex). All writes atomic.FA_diff_files,FA_diff_dirs,FA_apply_patch— newapply_text_patch()driver with strict hunk-context verification;diff_dirs_summary()wrapper returns plain dict for JSON-friendly payloads.FA_csv_filter(column projection + where clause),FA_csv_to_jsonl,FA_jsonl_iter(with limit),FA_jsonl_append.TextOpsException,DataOpsException.Batch B — deps + structured data + tracing (commit 5d7eeed)
FA_yaml_get/set/deletewith dotted-path access viayaml.safe_load(neveryaml.load).FA_parquet_read(with limit + column projection),FA_parquet_write,FA_csv_to_parquet.init_tracing()installs a globalTracerProviderwith a configurableSpanExporter(defaults to a no-op).action_span()context manager is a zero-cost no-op until initialised.ActionExecutor._execute_eventwraps every action in anautomation_file.actionspan tagged withfa.action=<name>.FA_tracing_initregistered.TracingException.PyYAML>=6.0,pyarrow>=15.0.0,opentelemetry-api>=1.25.0,opentelemetry-sdk>=1.25.0.Batch C — OneDrive + Box backends (commit 81737e5)
remote/onedrive/):OneDriveClientwraps arequests.Sessionwith MSAL-powered auth; offers bothlater_init(access_token)anddevice_code_login(client_id, tenant_id=None). Ops:FA_onedrive_upload_file/upload_dir(4 MiB simple-upload cap) /download_file/delete_item/list_folder/close. Graph paginations (@odata.nextLink) are followed.remote/box/):BoxClientwrapsboxsdk.OAuth2 + Client. Ops:FA_box_upload_file(returns new file id) /upload_dir(flattens tree — Box requires pre-existing folder ids) /download_file/delete_file/delete_folder(with recursive flag) /list_folder.OneDriveTab,BoxTab) + plugged into Transfer sidebar and Home status probes.OneDriveException,BoxException.msal>=1.28.0,boxsdk>=3.14.0.New dependencies (all required, not extras)
FA_yaml_*ops (safe_load only)FA_parquet_*opsTest plan
ruff check automation_file/ tests/ruff format --check automation_file/ tests/pytest tests/— 739 passed, 8 skippedFollow-ups (not in this PR)