feat: bing ads bulk upload (#3371)

* feat.bing ads first commit

* feat : interface flow for bingAds primary flow

* feat: adding marketo via interface

* fix: deleting type.go

* fix: cleaning up the code

* fix: adding with decided interface

* fix: addressing primary review comments

* fix: addressing code reviews

* fix: poll function update

* initial commit with bingAds

* implement fatchFailedEvents function placeholder

* feat(brt): onboarding bing ads audience bulk upload poll function (#3429)

* fix: adding temporary commits

* feat: initial commit for poll function

* fix: removing commented out code

* fix: small update in poll function

* feat: fetch failed results for bingAds

* fix: small edit in function definition

* fix: shortening function parameters fot fetchFailedEvents

* update message structure of bingads_audience (#3444)

* fix: adding success key logic and addressing review comments

* fix: making poll url an optional field

* fix: fixing in the status code data type

* fix: fixing rudder-auth service connection and editing test cases(#3455)

* update message structure of bingads_audience

* add implementation of tokenSource interface

* update existing testcases

* fix: fixing marketo bulk upload struct

* fix: small comments added

* fix: removing the poll status file after the response is written

* fix: adding function comments

* fix: remove the zip file after data upload

---------

Co-authored-by: Sudip Paul <67197965+ItsSudip@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* Update router/batchrouter/asyncdestinationmanager/bing-ads/bingads.go

Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* Update router/batchrouter/asyncdestinationmanager/bing-ads/bingads.go

Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* Update router/batchrouter/asyncdestinationmanager/bing-ads/bingads.go

Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* Update router/batchrouter/asyncdestinationmanager/bing-ads/bingads.go

Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* fix: review comment addressed

* remove magic number and use constant

* Apply suggestions from code review

Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* chore: refactors, and test cases added from chore.bingads

* fix: adding temporary commits

* feat: initial commit for poll function

* fix: removing commented out code

* fix: small update in poll function

* feat: fetch failed results for bingAds

* fix: small edit in function definition

* fix: shortening function parameters fot fetchFailedEvents

* update message structure of bingads_audience

* update message structure of bingads_audience (#3444)

* fix: adding success key logic and addressing review comments

* fix: making poll url an optional field

* fix: fixing in the status code data type

* add implementation of tokenSource interface

* update existing testcases

* fix: fixing rudder-auth service connection and editing test cases(#3455)

* update message structure of bingads_audience

* add implementation of tokenSource interface

* update existing testcases

* fix: fixing marketo bulk upload struct

* fix: small comments added

* fix: removing the poll status file after the response is written

* fix: adding function comments

* fix: remove the zip file after data upload

* add initial testcases

* feat(brt): adding unit test cases for bingads audience bulk upload (#3462)

* update message structure of bingads_audience

* add implementation of tokenSource interface

* update existing testcases

* add initial testcases

* add testcases

* refactor code and add testcases

* feat: adding primary unit test case for poll function (#3471)

* feat: adding primary unit test case for poll function

* feat: fetch failed events and poll functions test cases added

* fix: cleaning up the code

---------

Co-authored-by: shrouti1507 <shrouti.gangopadhyay@gmail.com>
Co-authored-by: shrouti1507 <60211312+shrouti1507@users.noreply.github.com>

* refer bingads sdk from other repo

* fix: retrieveImportantKey is deleted for bingads bulk upload and used better names for variables and structures(#3483)

* fix: retrieveImportantKey is deleted

* fix: changing variable names to more readble ones

* fix: changing variable names to more readble ones

* fix: adding interface for util function and mocking test cases

* fix go.mod

* fix: bingads unit test file location change and minor refactors

* update failedreason for bingads

* fix: removing unnecessary files

* fix: removed mock for util functions and parsed the code (#3512)

* feat: initial try for bingads test case update

* feat: test case update for poll and getUploadStatus

* feat: test case update for upload

* feat: test cas for new manager

* feat: removing unnecessary files

* fix: removing absolute paths

* fix: adding logger inits

* fix: adding test cases for upload  size control

* fix: uncomment the required code for uploadInterval

* fix: making temp files in test cases through env var

* fix: upadating the batch router test case by sending destination object as full

* fix: fixing go.mod file, removing panic and adding test case

* chore: bing ads tests

* fix: adding stats for bingads upload and upload status

* fix: omit panics for marketo

* fix: stats type

* chore: altering upload freq for QA

* fix: fix potential bug and add clientID logic

* fix: lint error fix

* fix: lint error fix

* fix: tidying up go.mod

* fix: tidying up go.mod

* feat: bingads audience adding support for multiple action polling and response parsing (#3565)

* feat: add support of remove and update in upload function of bingAds

* feat: adding support for multiple action polling and response parsing

* fix: pulling upload function updates as well as some bug fixes

* address comments

* update test cases

* refactor: refactoring bingads poll

* fix: fixing test cases error

* fix: adding comments

---------

Co-authored-by: ItsSudip <sudip.paul1997@gmail.com>

* refactor: run go mod tidy

* refactor: bingads uploader into multiple files

* feat: bingads destination config refresh (#3577)

* feat: add support of destination config refresh

* refactor: bingads implementation (#3573)

* refactor: bingads implementation

* fix: primary edits

* fix: removed ResultFileUrl field from structures

* fix: addressing code commnets

* fix: editing cumulative poll response

* fix: comment address part 1

* fix: comment address part 2

* refactor: fix issues

* fix: util changes

* fix: util changes

* refactor: createActionFile

* refactor clientID and Metadata

* refactor clientID and Metadata

* refactor names

* fix: bug solve

* fix action type

* fix: merged struct fields

* fix test cases

* fix: test cases part 1

* change TestBingAdsGetUploadStats

* fix: test cases part 2

* fix: remove unnecessary file

* fix: small fix

* address comments

* fix errors found while testing

* fix: remove commented code

---------

Co-authored-by: shrouti1507 <shrouti.gangopadhyay@gmail.com>
Co-authored-by: shrouti1507 <60211312+shrouti1507@users.noreply.github.com>
Co-authored-by: ItsSudip <sudip.paul1997@gmail.com>

* fix: addressing some lint issues

* fix: action file processing for event count zero

* fix: status count via loop

* fix: adding debug log

* fix: linting issues

* Fix: TestRouterManager

* fix: addressing comments regarding bingads

* fix: addressing comments regarding marketo

* add log on error

* add statuscode in log

* fix: Adding stat for non initializing destination

* fix initializtion of logger

* refactor: AsyncDestinationStruct

* fix: small fix in refresh dest

* refactor: initAsyncDestinationStruct

* refactor: initAsyncDestinationStruct

* refactor: handle_lifecycle setup

* fix: edit for testing

* Apply suggestions from code review

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* refactor: batch router handle async and review comments address (#3623)

* refactor: batch router handle async

* fix: pr comments

* fix: pr comments

* fix: pr comments

* fix: errors and address comments

* fix: remove unncessary lines

* fix: making upload time out 2 mins for testing scenarios

* fix: editing the stat type

* fix: editing invalid manager error message

---------

Co-authored-by: shrouti1507 <60211312+shrouti1507@users.noreply.github.com>
Co-authored-by: shrouti1507 <shrouti.gangopadhyay@gmail.com>

* fix: putting upload timeout to 30 mins

* Update router/batchrouter/handle_async.go

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* Update router/batchrouter/handle_async.go

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* Update router/batchrouter/handle_async.go

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* Update router/batchrouter/handle_async.go

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* fix: review comments addressed

* chore: comments addressed

* chore: comments addressed

* chore: comments addressed

* Apply suggestions from code review part 1

Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* fix: bingAds related new review comments

* fix: apply suggestions for marketo bulk upload from code review

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* Apply suggestions for marketo from code review part 2

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* fix: editing some import problem

* fix: lint error

* fix: pollAsyncStatus to handle error cases

* rename test files dir

* fix: move function to type.go

* Apply suggestions from code review

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* Apply name suggestions from code review

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* Apply suggestions from code review part 3

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>

* Apply name suggestions from code review part 3

Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>
Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>

* fix: code error edits

* fix: review comment for marshal unmarshal addressed

* fix: actionFile template making review

* fix: code error solve

* remove empty string check

* fix: bingAds SDK version change

* refactor: handle async

* chore: fix lint issues

* fix: marketo bulk upload poll function (#3684)

* Update router/batchrouter/asyncdestinationmanager/bing-ads/bulk_uploader.go

Co-authored-by: Dilip Kola <33080863+koladilip@users.noreply.github.com>

* fix: review comments addressed

* fix: remove unnecessary code

* fix: batch router attemptnos (#3696)

* fix: initial commit

* chore: clean up

* chore: clean up

* chore: clean up

* chore: clean up

---------

Co-authored-by: shrouti1507 <shrouti.gangopadhyay@gmail.com>
Co-authored-by: Chandra shekar Varkala <chandra@rudderlabs.com>

* update buffer size of scanner

* update buffer size of scanner to 50mb

* fix: batch router attemptnos, reports, rsources stats (#3698)

* chore: lint fixes

* chore: live events

* chore: max file size

---------

Co-authored-by: Chandra shekar Varkala <chandra@rudderlabs.com>
Co-authored-by: ItsSudip <sudip.paul1997@gmail.com>
Co-authored-by: Sudip Paul <67197965+ItsSudip@users.noreply.github.com>
Co-authored-by: Leonidas Vrachnis <leo.al.vra@gmail.com>
Co-authored-by: Dilip Kola <kdilipkola@gmail.com>
Co-authored-by: Dilip Kola <33080863+koladilip@users.noreply.github.com>
Co-authored-by: Rohith BCS <rohith.bcs@gmail.com>
Co-authored-by: chandumlg <54652834+chandumlg@users.noreply.github.com>

go.mod

@@ -234,6 +234,7 @@ require (
 	github.com/rivo/uniseg v0.1.0 // indirect
 	github.com/rs/xid v1.5.0 // indirect
 	github.com/rs/zerolog v1.28.0 // indirect
+	github.com/rudderlabs/bing-ads-go-sdk v0.2.0
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/segmentio/backo-go v1.0.1 // indirect
 	github.com/shirou/gopsutil/v3 v3.23.4 // indirect

go.sum

@@ -1740,6 +1740,8 @@ github.com/rs/zerolog v1.28.0 h1:MirSo27VyNi7RJYP3078AA1+Cyzd2GB66qy3aUHvsWY=
 github.com/rs/zerolog v1.28.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0=
 github.com/rudderlabs/analytics-go v3.3.3+incompatible h1:OG0XlKoXfr539e2t1dXtTB+Gr89uFW+OUNQBVhHIIBY=
 github.com/rudderlabs/analytics-go v3.3.3+incompatible/go.mod h1:LF8/ty9kUX4PTY3l5c97K3nZZaX5Hwsvt+NBaRL/f30=
+github.com/rudderlabs/bing-ads-go-sdk v0.2.0 h1:Q9J7uCtMsqMo5106wIFenFoNX8slDL3ZmsYy9Nm2wus=
+github.com/rudderlabs/bing-ads-go-sdk v0.2.0/go.mod h1:n718Q+g1OBGi7rOIRLvKU+1Ep0gwkEOdQzMdwvzpCXU=
 github.com/rudderlabs/compose-test v0.1.2 h1:GTt8meEXkdOXPZWKW0r5mDRA5ddachR/pR+xsrWNU54=
 github.com/rudderlabs/compose-test v0.1.2/go.mod h1:J1UaMZXoXRGHE0+VLpHZ/0hp+1F2m5cmjOKcwaN5LOU=
 github.com/rudderlabs/parquet-go v0.0.2 h1:ZXRdZdimB0PdJtmxeSSxfI0fDQ3kZjwzBxRi6Ut1J8k=