bugfix/evancole-untitled-casting update

[fivetran-joemarkiewicz]

Are you a current Fivetran customer?

Fivetran created PR, but takes the bulk of the work from PR #24 that was opened by @evancole-untitled.

What change(s) does this PR introduce?

Did you update the CHANGELOG?

• Yes

Does this PR introduce a breaking change?

• Yes (please provide breaking change details below.)
• No (please provide an explanation as to how the change is non-breaking below.)

Since this is such a large update that changes the datatypes of multiple fields that could be used in downstream join conditions, this should be a breaking change.

Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)

• Yes

Is this PR in response to a previously created Bug or Feature Request

• Yes, Issue/Feature [Bug] dbt/quickbooks package breaking on build due to non-numerical sales_item_item_id #23
• No

How did you test the PR changes?

• CircleCi
• Local (please provide additional testing details below)

Select which warehouse(s) were used to test the PR

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Provide an emoji that best describes your current mood

💸

Feedback

We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your feedback on our existing dbt packages or what you'd like to see next.

[fivetran-joemarkiewicz]

@evancole-untitled would you be able to try the following package dependency and see if the issue still persists?

packages:
  - git: https://github.com/fivetran/dbt_quickbooks.git
    revision: bugfix/evancole-untitled-casting
    warn-unpinned: false

[fivetran-joemarkiewicz]

@evancole-untitled I was able to finally get the package to compile! 🎉

I actually ended up needing to cast all of the intentional type_int fields to type_string in order for the models to properly compile. I actually believe that this is the safest route to go to ensure the package is stable into the future. If you had an ID field that was a string, then the odds of this happening for other ID fields is extremely likely.

Would you be able to try the working branch of the modeling package and let me know if you still see the issue persist?

packages:
  - git: https://github.com/fivetran/dbt_quickbooks.git
    revision: bugfix/evancole-untitled-casting
    warn-unpinned: false

[evancole-untitled]

It almost ran! I crashed out in the quickbooks package with this error

19:23:42 Completed with 1 error and 0 warnings:
19:23:42
19:23:42 Database Error in model quickbooks__general_ledger_by_period (models/quickbooks__general_ledger_by_period.sql)
19:23:42 failed to find conversion function from "unknown" to character varying
19:23:42 compiled SQL at target/run/quickbooks/models/quickbooks__general_ledger_by_period.sql
19:23:42
19:23:42 Done. PASS=109 WARN=0 ERROR=1 SKIP=2 TOTAL=112

[fivetran-joemarkiewicz]

@evancole-untitled hmmm I wonder why it is passing on my side 🤔

If you look into the compiled output of the model with the error, are you able to see which field is causing the issue on your end?

[evancole-untitled]

i suspect it's this block down at the very bottom:

final as ( select * from general_ledger_balances union all select * from retained_earnings )

As for why it worked for you and not for me, my first guess would be that you're using testing it using a grownup warehouse like Snowflake, which will attempt to do typing and promotions during unions or joins on complex queries, instead of Redshift, which will attempt to ridicule you :/ I tested this out with this code:

WITH t1 AS (SELECT 'text' as c1) SELECT c1 FROM t1 INNER JOIN t1 AS t2 USING (c1)

It runs just fine in Snowflake, but in Redshift I get the same "failed to find conversion function from "unknown" to character varying" error.

…

On Fri, Mar 25, 2022 at 4:09 PM Joe Markiewicz ***@***.***> wrote: @evancole-untitled <https://github.com/evancole-untitled> hmmm I wonder why it is passing on my side 🤔 If you look into the compiled output of the model with the error, are you able to see which field is causing the issue on your end? — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVERHMFPO3D5SPZIA7CVU2TVBYMRHANCNFSM5RSAUUZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fivetran-joemarkiewicz]

Thanks @evancole-untitled!

I wonder if the constant expressions created within the int_quickbooks__retained_earnings model is causing this issue within redshift. I just pushed some changes to the branch the add explicit casting to hopefully resolve this "unknown" issue. Would you mind trying the branch again and let me know if you still see the issue?

Also, would you be able to try and dbt run --full-refresh just as a check to ensure there isn't an error from a previous run that is causing the hangup.

[evancole-untitled]

Sure! I can confirm that the error persists even with the --full-refresh flag.

…

On Mon, Mar 28, 2022 at 11:35 AM Joe Markiewicz ***@***.***> wrote: Thanks @evancole-untitled <https://github.com/evancole-untitled>! I wonder if the constant expressions created within the int_quickbooks__retained_earnings model is causing this issue within redshift. I just pushed some changes to the branch the add explicit casting to hopefully resolve this "unknown" issue. Would you mind trying the branch again and let me know if you still see the issue? Also, would you be able to try and dbt run --full-refresh just as a check to ensure there isn't an error from a previous run that is causing the hangup. — Reply to this email directly, view it on GitHub <#25 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVERHMGQ2YMPG6CEWKT5THDVCHGUHANCNFSM5RSAUUZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fivetran-joemarkiewicz]

Darn 😞

I am a bit stuck since I have tested this on my own Redshift instance and am unable to replicate the error. I think the next best step if the latest changes didn't resolve the issue would be to setup some time during out office hours to dig into this together live.

[fivetran-joemarkiewicz]

It was found that the latest update to this branch works for @evancole-untitled. This is ready for review!

[fivetran-sheringuyen]

Heya @fivetran-joemarkiewicz, great PR, that was quite a lot of changes indeed! I will need your help getting access the the QB data to test the models, but I double checked all the macros and it looks like you've got all the ID columns specified as datatype string.

I left a couple of comments regarding the final models -- I'd like to understand the logic that you followed to cast certain ID columns and not all of them so that I can properly review that portion of the PR.

Let me know if you have any questions!

[fivetran-sheringuyen]

@fivetran-joemarkiewicz dbt test ran great, however dbt test had one failure (that is also the failure that is encountered for the modeling pkg due to source test failure) -- is this expected?

18:33:30  
18:33:30  Failure in test not_null_stg_quickbooks__bill_linked_txn_bill_payment_id (models/stg_quickbooks.yml)
18:33:30    Got 2 results, configured to fail if != 0
18:33:30  
18:33:30    compiled SQL at target/compiled/quickbooks_source/models/stg_quickbooks.yml/not_null_stg_quickbooks__bill_linked_txn_bill_payment_id.sql
18:33:30  
18:33:30  Done. PASS=56 WARN=0 ERROR=1 SKIP=0 TOTAL=57```

[fivetran-joemarkiewicz]

@fivetran-sheringuyen I just looked at the source data and this seems to be a case where the test failed successfully. Looking at the source data there are in fact nulls, but that is not expected. This is a strange case where the package highlighted a data quality issue properly.

This will be something we want to investigate within the source data, but this is expected behavior for the package.

[fivetran-sheringuyen]

lgtm!