Cannot backfill raw table because of error "Cannot read property 'forEach' of undefined"

[XaviePaez]

[REQUIRED] Step 2: Describe your configuration

• Extension name: firestore-bigquery-export
• Extension version: 0.1.27

[REQUIRED] Step 3: Describe the problem

Running the @firebaseextensions/fs-bq-import-collection import scripts leads to an inserting error while trying to backfill a specific collection.

This error occurs only for one collection and for the project labelled "prod", the others have not faced this issue. Indeed, we have 2 projects implemented to validate "Staging" and "PROD" environments and this error is only appearing on the PROD project for this specific collection and no data is uploaded during the import scripting (except the one from the Stream Collection to BigQuery plugged).

Steps to reproduce:

• Run the @firebaseextensions/fs-bq-import-collection.

Logs Staging :

Importing data from Cloud Firestore Collection: rewards, to BigQuery Dataset: project_staging_firestore_export, Table: rewards_raw_changelog
{"severity":"INFO","message":"BigQuery dataset already exists: project_staging_firestore_export"}
{"severity":"INFO","message":"BigQuery table with name rewards_raw_changelog already exists in dataset project_staging_firestore_export!"}
{"severity":"INFO","message":"Clustering removed on rewards_raw_changelog"}
{"severity":"WARNING","message":"No valid table reference is available. Skipping partitioning"}
{"severity":"WARNING","message":"Cannot partition an existing table project_staging_firestore_export_rewards_raw_changelog"}
{"severity":"INFO","message":"View with id rewards_raw_latest already exists in dataset project_staging_firestore_export."}
{"severity":"INFO","message":"Updated 'rewards_raw_latest' table with a 'document_id' column"}
{"severity":"INFO","message":"Inserting 153 row(s) of data into BigQuery"}
{"severity":"INFO","message":"Inserted 153 row(s) of data into BigQuery"}
---------------------------------------------------------
Finished importing 153 Firestore rows to BigQuery
---------------------------------------------------------

Logs PROD :

Importing data from Cloud Firestore Collection: rewards, to BigQuery Dataset: project_prod_firestore_export, Table: rewards_raw_changelog
{"severity":"INFO","message":"BigQuery dataset already exists: project_prod_firestore_export"}
{"severity":"INFO","message":"BigQuery table with name rewards_raw_changelog already exists in dataset project_prod_firestore_export!"}
{"severity":"INFO","message":"Clustering removed on rewards_raw_changelog"}
{"severity":"WARNING","message":"No valid table reference is available. Skipping partitioning"}
{"severity":"WARNING","message":"Cannot partition an existing table project_prod_firestore_export_rewards_raw_changelog"}
{"severity":"INFO","message":"View with id rewards_raw_latest already exists in dataset project_prod_firestore_export."}
{"severity":"INFO","message":"Updated 'rewards_raw_latest' table with a 'document_id' column"}
{"severity":"INFO","message":"Inserting 22090 row(s) of data into BigQuery"}
{"severity":"INFO","message":"Retried to insert 22090 row(s) of data into BigQuery (ignoring unknown columns)"}
{"severity":"INFO","message":"Inserting 22090 row(s) of data into BigQuery"}
{"severity":"WARNING","message":"Error when inserting data to table."}
Error importing Collection to BigQuery: TypeError: Cannot read property 'forEach' of undefined

Expected result

Same result than Staging environment

Actual result

{"severity":"WARNING","message":"Error when inserting data to table."}
Error importing Collection to BigQuery: TypeError: Cannot read property 'forEach' of undefined

[cabljac]

potentially related/fixed by #1292

[XaviePaez]

Hello @cabljac ,

Thank you for your quick comment.

Do you know what could be the origin of the issue ? And when this would be fixed ?

Thank you in advance for your help.

[cabljac]

I believe it's this line, parsed.errors is not defined in this case. Seems to be different to the above fix actually

[XaviePaez]

Thank you @cabljac.
Do I have to modify something ? or a git pull will solve the issue ?

[cabljac]

I've opened a PR to fix this issue, so it should be fixed in the next release. In the mean time if it's urgent you could patch your version, essentially i think we need optional chaining on line 253:

parsed.errors?.forEach((e) => console.error(....

[dackers86]

Closing as issue resolved in #1310 and released in version 0.1.29