Sm/qa-jsforce-3-bulk by mshanemc · Pull Request #843 · salesforcecli/plugin-data

mshanemc · 2024-02-27T18:30:44Z

originally created to QA for @W-14996843@

What does this PR do?

continued refactoring work for jsforce3 (esp async query to not return results or check even once)
use jsforce3 bulk2 polling events
~~allow --use-tooling-api with --bulk on query (Tooling API and Bulk API should not be mutually exclusive for data queries forcedotcom/cli#2667)~~
reorganize the reporters, mostly using functions. If I created a bug, it's somewhere in there. split them into different files, along with their tests
more UT for reporter logic
1 nut fix where bulk delete resume human output wasn't being tested
drop support for "sfdx" and ":" in help/message output. only sf and spaces
improve the perf and reduce API call usages for monitoring bulk jobs
improve the error result if you send an uparseable csv file
improve the output of the bulk job event updates
remove the unused chai-as-promised devDep

Job a07EE00001l5rHsYAI Status Job Complete Records processed 500 Records failed 0. =>
Job a07EE00001l5rHsYAI | Status Job Complete | Records processed 500 | Records failed 0.

there are some // ts-expect-error because jsforce-node.Connection isn't exactly identical to sfdx-core.Connection that extends jsforce v2

one obvious difference is that they have a different implementation of conn.bulk/.bulk2 but there are others about private members (ex: sfdx-core inserts its own logger)

When jsforce-node is rolled out in sfdx-core and sfCommand those will start failing, hopefully, and we can remove them

@W-15141145@

mshanemc · 2024-03-01T19:41:22Z

src/bulkUtils.ts

  }
 };
-
-export const waitOrTimeout = async (job: IngestJobV2<Schema, IngestOperation>, wait: number): Promise<void> => {


jsforce now handles the polling/timeout logic and emits events, so this is removed

mshanemc · 2024-03-01T19:42:42Z

src/commands/data/query.ts

          );
      if (!this.jsonEnabled()) {
-        displayResults({ ...queryResult }, flags['result-format'] as FormatTypes);
+        displayResults({ ...queryResult }, flags['result-format']);


uses the new option flag that types work correctly with

mshanemc · 2024-03-01T19:43:39Z

src/commands/data/query.ts

-    connection.bulk2.pollTimeout = timeout.milliseconds ?? Duration.minutes(5).milliseconds;
-    let res: Record[];
+    if (timeout.milliseconds === 0) {
+      const job = new QueryJobV2(


this would sometimes have results if the query finished quickly. This way it does what the user asked (open and don't be checking) and returns sooner.

mshanemc · 2024-03-01T19:43:51Z

src/commands/data/query.ts

    try {
-      res = (await connection.bulk2.query(query, allRows ? { scanAll: true } : {})) ?? [];
-      return transformBulkResults(res, query);
+      connection.bulk2.pollTimeout = timeout.milliseconds ?? Duration.minutes(5).milliseconds;


now handling the unintentional timeout

mshanemc · 2024-03-01T19:44:28Z

src/commands/data/query/resume.ts


  public static readonly flags = {
-    'target-org': { ...optionalOrgFlagWithDeprecations, summary: queryMessages.getMessage('flags.targetOrg.summary') },
+    'target-org': optionalOrgFlagWithDeprecations,


use the standard summary!

mshanemc · 2024-03-01T20:15:59Z

test/commands/data/query.test.ts

-            stdout
-          )
-        ).to.be.true;
+        expect(stdout).to.match(


provides nicer output than expected false to be true

mshanemc · 2024-03-01T20:17:14Z

test/test-files/data-project/data/bulkUpsertLarge.csv

I think it's handy to have on of these for QA so I left it in.

cristiand391 · 2024-03-18T16:07:21Z

src/queryUtils.ts

+  // eslint-disable-next-line no-console
+  console.error(i);
+  return i;
+};


this isn't used in the plugin.

yeah, I'll take it out. I probably need to put it in kit.

cristiand391 · 2024-03-20T22:14:35Z

QA notes:

data query
✅ nested fields are properly rendered
✅ csv reporter outputs valid csv, tested by building csv for record deletion
✅ json reporter
❌ respects wait time with bulk query

--wait is specified in minutes, no matter what value I provide in debug logs the query polling uses the default 30_000ms (30s)

JSFORCE_LOG_LEVEL=DEBUG ./bin/run.js data query --query "SELECT Id, Name,FROM Account" --bulk --wait 10

❌ can do a bulk query on tooling objects
At first I thought it worked when querying ApexClass but that's available outside tooling, after reading the docs linked in the GH discussion I see this requires adding support for the tooling endpoint to do bulk queries on jsforce, see:
https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling/tooling_api_objects_metadatacomponentdependency_bulk2_usage.htm

IMO we shouldn't implement that yet, mostly because:

only MetadataComponentDependency seems to support tooling bulk queries
it's been in beta for a few years and doesn't seem to be a priority (see group info: https://trailhead.salesforce.com/trailblazer-community/groups/0F93A00000020Vu?tab=discussion&sort=LAST_MODIFIED_DATE_DESC)

✅ bad soql query still shows error from API on --bulk
✅ --wait 0 or --async query, can do query resume and see a 1000ms timeout instead of 0.

data upsert bulk
❌ no update on successful/unprocessed/failed records
also couldn't find the polling logs when running this:

JSFORCE_LOG_LEVEL=DEBUG ./bin/dev.js data upsert bulk --file test/test-files/data-project/data/bulkUpsertLarge.csv --external-id Id --sobject Account --wait 10

seems using the bulk2 from conn instead of the one from jsforce-node, that explains the missing events for the indicator

data delete bulk
❌ no update on successful/unprocessed/failed records

same as upsert bulk

mshanemc · 2024-03-21T12:31:37Z

IMO we shouldn't implement that yet,

Yeah, I agree. I'll take that stuff out.

cristiand391 · 2024-03-21T15:15:07Z

QA update:

❌ respects wait time with bulk query

✅ fixed, passed multiple values in min and jsforce gets those in ms

❌ can do a bulk query on tooling objects

✅ --bulk and --use-tooling-api are exclusive again

data upsert bulk

❌ no update on successful/unprocessed/failed records

✅ fixed, command catches inProgress events and updates the spinner on each poll

❌ respects wait time
testes multiple values with --wait, jsforce ends up with a big timeout on polling and only exits on finish or Ctrl-c.

JSFORCE_LOG_LEVEL=DEBUG ./bin/run.js data upsert bulk --file test/test-files/data-project/data/bulkUpsertLarge.csv --external-id Id --sobject Account --wait 10

JSFORCE_LOG_LEVEL=DEBUG ./bin/run.js data upsert bulk --file test/test-files/data-project/data/bulkUpsertLarge.csv --external-id Id --sobject Account --wait 1

spinner status after 2m

data delete bulk

❌ no update on successful/unprocessed/failed records

✅ fixed

❌ respects wait time

same as upsert bulk

cristiand391 · 2024-03-21T16:14:27Z

QA notes:

data delete bulk and data upsert bulk:

❌ respects wait time

✅ fixed
✅ print message if to org open in org if a record failed
✅ exits after 1m on --wait 1 and big bulk job
✅ exit code !=0 on record failures

mshanemc added 29 commits February 27, 2024 11:24

chore: use default flag messages

08bc950

chore: don't support dynamic bin/separator

9c0490d

refactor: correct async stream listeners, more functions

3af0705

refactor: oclif option flag

cd94e94

refactor: minor flags, batcher, v2 type changes

27507a4

refactor: v2 bulk types, prune base command

2f8b73f

fix: don't send 0 polling timeout

97e323a

test: all nuts passing

51958e4

refactor: pruning base command and messages

7049dcb

refactor: remove job from base

8c973fe

refactor: functions and nut fix

8df23c0

chore: partial refactor of reporters

3bf3fda

refactor: split humanReporter

a0787df

refactor: extract csvReporter

42f7c06

refactor: humanReporter

0f4d057

refactor: csvReporter with more UT testability

a45c00e

style: nicer results banner

651725d

test: handy file for manual QA use

2ad8f94

feat: use jsforce polling inProgress event

fcd9a3e

feat: allow tooling with bulk

0e4183a

refactor: remove more dynamic "sfdx" and ":" in messages

e457f51

test: remove only

675a0af

test: nut change for output change

02e4e38

style: formatting

3fd5223

chore: back to regular jsforce

2d94c5f

fix: do json last

8c54d70

chore: use jsforce-node

d59076a

feat: return after job open for async query

3b7780b

chore: lockfile

bc8dd6e

mshanemc commented Mar 1, 2024

View reviewed changes

mshanemc added 3 commits March 1, 2024 16:18

refactor: generic tuple/object

7bf7903

test: testing subquery logic

20871a6

refactor: slit reporter tests to match code

f3a2ff4

mshanemc marked this pull request as ready for review March 1, 2024 22:48

mshanemc requested a review from a team as a code owner March 1, 2024 22:48

mshanemc added 2 commits March 1, 2024 16:49

chore: restore tsconfigs

20ece2e

chore: dep cleanup

9e47fcf

cristiand391 reviewed Mar 18, 2024

View reviewed changes

cristiand391 approved these changes Mar 18, 2024

View reviewed changes

mshanemc mentioned this pull request Mar 20, 2024

fix: support baseFlags on SfCommand #863

Closed

mshanemc added 2 commits March 20, 2024 11:06

Merge remote-tracking branch 'origin/main' into sm/qa-jsforce-3-bulk

1a7dff0

chore: remove log util

f51037a

mshanemc added 2 commits March 21, 2024 07:22

chore: respects wait time with bulk query

c288317

refactor: construct a bulk2 instead of using the one from Connection

d93b682

mshanemc added 2 commits March 21, 2024 07:42

refactor: restore exclusivity

6382014

refactor: spinner output

554a5ad

chore: parenetheses matter for ??

98df3de

cristiand391 merged commit 5bc7157 into main Mar 21, 2024

cristiand391 deleted the sm/qa-jsforce-3-bulk branch March 21, 2024 16:16

cristiand391 mentioned this pull request Jun 3, 2024

bulk and use-tooling-api cannot be used in the same operation forcedotcom/cli#2900

Closed

Conversation

mshanemc commented Feb 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cristiand391 commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mshanemc commented Mar 21, 2024

Uh oh!

cristiand391 commented Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cristiand391 commented Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mshanemc commented Feb 27, 2024 •

edited

Loading

cristiand391 commented Mar 20, 2024 •

edited

Loading

cristiand391 commented Mar 21, 2024 •

edited

Loading

cristiand391 commented Mar 21, 2024 •

edited

Loading