Fix landings stream to be incremental

[loeakaodas]

Description of change

• Fix landings stream to be incremental as per TDL-16356
• closes Revert "Fix landings pagination (#48)" #56
• closes Stitch/Typeform connexion turn into full-synchronization  #54

Manual QA steps

• ran tap locally
• ran against singer-check-tap and target-stitch --dry-run
• ran tap-tester locally
• tests passing in circle

Risks

• minimal, the stream will now be incremental

Rollback steps

• revert this branch

[luandy64]

Do we know that submitted_at and landed_at are always the same? If not, then shouldn't the replication key be submitted_at?

[loeakaodas]

They are definitely not the same, at least in the test data submitted_at is always greater than landed_at. I mimicked the setup of the answers stream for making the landings stream incremental and used landed_at as the replication key. This was because both streams use get_form_responses method from http.py to call the API.

IDS.LANDINGS: {"replication_method": "INCREMENTAL", "replication_keys": ["landed_at"]},
IDS.ANSWERS: {"replication_method": "INCREMENTAL", "replication_keys": ["landed_at"]},

I'm not sure why landed_at was chosen for the answers stream, but maybe there was a case where it would be null if a form isn't submitted, possibly? The answers schema doesn't even have submitted_at as one of it's field.

To keep it consistent with the tap's behavior on the answers stream I think we should leave it as is, but I did test it with tap-tester locally and all tests were passing if I switched the landings stream to use submitted_at as the replication key.

A more robust solution would be to switch both the landings and answers stream to use submitted_at as the replication key given the API limitations when using the after query param. This would require a schema change for the answers stream and might run into data issues if submitted_at can be null.

[MarcosPala]

Any status on this?

[loeakaodas]

Any status on this?

@KrisPersonal @luandy64 could this be merged? it fixes an issues for clients in Stitch and is passing tests.