Unable to make the incremental model work with insert_overwrite strategy

[adithyapathipaka]

Describe the bug

I'm having trouble executing models with materialzed=incremental in dbt with glue spark. Incremental condition is not adding to sql query in is_incremental block.

Profile:

aws_demo_proj:
  target: aws_dev
  outputs:
    aws_dev:
      type: glue
      query-comment: This is a glue dbt example
      role_arn: arn:aws:iam::xxxxxxxx:role/dbt-demo-role
      region: eu-central-1
      workers: 5
      worker_type: G.2X
      schema: "det"
      database: "det_landing_data_platform"
      session_provisioning_timeout_in_seconds: 120
      location: "s3a://det-landing-data-lake"
      idle_timeout: 120

Incremental Model Config and SQL:

{{ config(
    materialized='incremental',
    custom_location='s3a://det-published-data-lake/booking-solutions/fulfilment_view_data',
    incremental_strategy='insert_overwrite',
    schema='published_booking_solutions',
    file_format='parquet',
    partition_by=['business_date']

) }}

SELECT new_process_id, booking_source,
((booked_restored_gross_price_hotel - new_restored_gross_price_hotel) * new_restored_gross_price_customer / new_restored_gross_price_hotel) as savings,
(booked_restored_gross_price_hotel * new_restored_gross_price_customer / new_restored_gross_price_hotel) as booked_restored_gross_price_customer,
new_restored_gross_price_customer,
booked_price_rate_type,
init_booking_multisourced,
rebooking_multisourced,
initial_crs_type,
DATE(concat_ws('-', year, month, day)) as business_date
FROM prod_landing_booking_solutions.fulfilment_view
WHERE
    -- REQUIRED TO OMIT DIVISION BY ZERO value of new_restored_gross_price_hotel
    new_restored_gross_price_hotel > 0
    AND new_process_id IS NOT NULL
    AND booked_restored_gross_price_hotel > new_restored_gross_price_hotel
    {% if is_incremental() %}
    and  DATE(concat_ws('-', year,month, day)) >= (select max(business_date) from  {{this}} )
    {% endif %}

Steps To Reproduce

Rerun the dbt model and check the query generated.

Expected behavior

Incremental conditions need to be added to the SQL query from the next of table creation.

Screenshots and log output

incremental_first_load.log
incremental_second_load.log

System information

The output of dbt --version:

Core:
  - installed: 1.1.1
  - latest:    1.1.1 - Up to date!

Plugins:
  - spark: 1.1.0 - Up to date!
  - glue: 0.2.0 - Up to date!

The operating system you're using: Mac Os

The output of python --version:

Python 3.8.8

Additional context

Add any other context about the problem here.

[adithyapathipaka]

Even tried with the Append strategy, it still doesn't work.
Below are the logs,
incremental_second_load_append.log
incremental_first_load_append.log

[adithyapathipaka]

#45
Hello team,
I have tried to debug the issue and found that is_incremental macro is get the table relation from impl.py as list instead of dict type/object type.

I have created a fix for it and available in PR 45