tt_search_api ! HTTP 504 Gateway Timeout

[vot4ntonio]

Hi, I've been trying to gather videos using the tt_search_api all day, but the initial request always ends up timing out.
The tt_user_info seems to be working.
The same code has worked in the past, so I don't think it's an issue with the query construction.
I reinstalled the package and changed the IP address, but the response is still "! HTTP 504 Gateway Timeout".

query_elezioni <- query() |>
  query_or(field_name = "hashtag_name",
           operation = "IN",
           field_values = c("#elezioni",  "#europee", "#elezionieuropee2024", "#europee2024","#elezionieuropee")) |>
  query_or(field_name = "keyword",
           operation = "IN",
           field_values = c("elezioni", "europee","europee2024", "elezionieuropee2024",  "elezionieuropee")) |>
  query_and(field_name = "region_code", operation = "IN", field_values = c("it", "IT"))

df <- tt_search_api(query = query_elezioni,
                                           start_date = as.Date("2024-05-07"),
                                           end_date = as.Date("2024-05-14"),
                                           fields = "all",
                                           max_pages = 100L,
                                           verbose = T, cache = T)

Response

Error in `httr2::req_perform()`:                               
! HTTP 504 Gateway Timeout.
• status: timeout
• message: Request timed out. Please try again
• log_id: #########################################
Run `rlang::last_trace()` to see where the error occurred.
✖ Making initial request [3m 3.5s]
> rlang::last_trace()
<error/httr2_http_504>
Error in `httr2::req_perform()`:
! HTTP 504 Gateway Timeout.
• status: timeout
• message: Request timed out. Please try again
• log_id: ################ Error in `httr2::req_perform()`:                               
! HTTP 504 Gateway Timeout.
• status: timeout
• message: Request timed out. Please try again
• log_id: #########################################
Run `rlang::last_trace()` to see where the error occurred.
✖ Making initial request [3m 3.5s]
> rlang::last_trace()
<error/httr2_http_504>
Error in `httr2::req_perform()`:
! HTTP 504 Gateway Timeout.
• status: timeout
• message: Request timed out. Please try again
• log_id: #########################################
---
Backtrace:
    ▆
 1. └─traktok::tt_search_api(...)
 2.   └─traktok:::tt_query_request(...)
 3.     ├─httr2::resp_body_json(...)
 4.     │ └─httr2:::check_response(resp)
 5.     │   └─httr2:::is_response(resp)
 6.     └─httr2::req_perform(...)
Run rlang::last_trace(drop = FALSE) to see 3 hidden frames.
> rlang::last_trace(drop = FALSE)
<error/httr2_http_504>
Error in `httr2::req_perform()`:
! HTTP 504 Gateway Timeout.
• status: timeout
• message: Request timed out. Please try again
• log_id: #########################################
---
Backtrace:
    ▆
 1. └─traktok::tt_search_api(...)
 2.   └─traktok:::tt_query_request(...)
 3.     ├─httr2::resp_body_json(...)
 4.     │ └─httr2:::check_response(resp)
 5.     │   └─httr2:::is_response(resp)
 6.     └─httr2::req_perform(...)
 7.       └─httr2:::handle_resp(req, resp, error_call = error_call)
 8.         └─httr2:::resp_abort(resp, req, body, call = error_call)
 9.           └─rlang::abort(...)3B200525BD2576705D163
---
Backtrace:
    ▆
 1. └─traktok::tt_search_api(...)
 2.   └─traktok:::tt_query_request(...)
 3.     ├─httr2::resp_body_json(...)
 4.     │ └─httr2:::check_response(resp)
 5.     │   └─httr2:::is_response(resp)
 6.     └─httr2::req_perform(...)
Run rlang::last_trace(drop = FALSE) to see 3 hidden frames.
> rlang::last_trace(drop = FALSE)
<error/httr2_http_504>
Error in `httr2::req_perform()`:
! HTTP 504 Gateway Timeout.
• status: timeout
• message: Request timed out. Please try again
• log_id: #########################################
---
Backtrace:
    ▆
 1. └─traktok::tt_search_api(...)
 2.   └─traktok:::tt_query_request(...)
 3.     ├─httr2::resp_body_json(...)
 4.     │ └─httr2:::check_response(resp)
 5.     │   └─httr2:::is_response(resp)
 6.     └─httr2::req_perform(...)
 7.       └─httr2:::handle_resp(req, resp, error_call = error_call)
 8.         └─httr2:::resp_abort(resp, req, body, call = error_call)
 9.           └─rlang::abort(...)

Version
platform x86_64-w64-mingw32
arch x86_64
os mingw32
crt ucrt
system x86_64, mingw32
status
major 4
minor 3.3
year 2024
month 02
day 29
svn rev 86002
language R
version.string R version 4.3.3 (2024-02-29 ucrt)
nickname Angel Food Cake

[vot4ntonio]

Hi, I am still unable to collect data but today the error message is different and it seems to relate with the error format.

Error in `httr2::req_perform()`:                               
! Failed to parse error body with method defined in `req_error()`.
Caused by error in `httr2::resp_body_json()`:
! Unexpected content type "text/html".
• Expecting type "application/json" or suffix "json".
Run `rlang::last_trace()` to see where the error occurred.
✖ Making initial request [4m 11.6s]
> rlang::last_trace()
<error/rlang_error>
Error in `httr2::req_perform()`:
! Failed to parse error body with method defined in `req_error()`.
Caused by error in `httr2::resp_body_json()`:
! Unexpected content type "text/html".
• Expecting type "application/json" or suffix "json".
---
Backtrace:
     ▆
  1. └─traktok::tt_search_api(...)
  2.   └─traktok:::tt_query_request(...)
  3.     ├─httr2::resp_body_json(...)
  4.     │ └─httr2:::check_response(resp)
  5.     │   └─httr2:::is_response(resp)
  6.     └─httr2::req_perform(...)
  7.       └─httr2:::handle_resp(req, resp, error_call = error_call)
  8.         └─httr2:::error_body(req, resp, error_call)
  9.           ├─rlang::try_fetch(...)
 10.           │ ├─base::tryCatch(...)
 11.           │ │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 12.           │ │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 13.           │ │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 14.           │ └─base::withCallingHandlers(...)
 15.           └─httr2:::req_policy_call(req, "error_body", list(resp), default = NULL)
 16.             ├─rlang::exec(req$policies[[name]], !!!args)
 17.             └─traktok (local) `<fn>`(`<httr2_rs>`)
 18.               ├─base::paste("status:", httr2::resp_body_json(resp)$error$code)
 19.               └─httr2::resp_body_json(resp)

[JBGruber]

That's very strange! I'm traveling today, so can't look into it. I would be curious what the error is. You should be able to get it with

httr2::last_response() |>
   httr2::resp_body_string() 

I'm typing on my phone, so check for typos.

[JBGruber]

I just tried to reproduce your error and ran into the same issue. It seems to me that the research API's search endpoint is currently experiencing issues. I used the example curl call from the docs directly to see what is happening:

curl -X POST \
  'https://open.tiktokapis.com/v2/research/video/query/?fields=id,like_count' \
  -H 'authorization: bearer clt.example12345Example12345Example' \
  -d '{ 
          "query": {
              "and": [
                   { "operation": "IN", "field_name": "region_code", "field_values": ["US", "CA"] },
                   { "operation": "EQ", "field_name": "keyword", "field_values": ["hello world"] }
               ]
          }, 
          "start_date": "20220615",
          "end_date": "20220628",
          "max_count": 10
}'

I replaced clt.example12345Example12345Example with a valid token and received:

<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>
An error occurred while processing your request.<p>
Reference&#32;&#35;221&#46;4a521302&#46;1716843035&#46;16707b1
<P>https&#58;&#47;&#47;errors&#46;edgesuite&#46;net&#47;221&#46;4a521302&#46;1716843035&#46;16707b1</P>
</BODY></HTML>

I contacted support and opened a ticket. You might want to do the same.

[vot4ntonio]

Thank you for your answer. I've already opened a ticket and I am waiting for their response.

[fabiogiglietto]

It is definitely an issue with the research API. I have been experiencing intermittent problems with this script, which is scheduled to run daily, for the past few weeks. I'll try adding my ticket to the queue, but I never heard back from them in my previous attempts

[JBGruber]

I can no longer reproduce this. At least at the moment.

Do you have more insights from your scheduled runs @fabiogiglietto? Also, very cool project. Nice to see you get good use out of traktok 😁 !

[fabiogiglietto]

I am still encountering this API issue; however, you may close this thread if you prefer, as it seems that the problem does not originate from Traktok itself. Have you heard back from TikTok about your ticket?

[vot4ntonio]

Unfortunately, after a first email in which I was told they would take over the issue, I received no further response.

[vot4ntonio]

Today I was able to collect a week of data. However, when I changed the timespan I received the following response:

> rlang::last_trace(drop = FALSE)
<error/httr2_http_400>
Error in `httr2::req_perform()`:
! HTTP 400 Bad Request.
• status: invalid_params
• message: Search Id 7378809778820092971 is invalid or expired
• log_id: #######################################
---
Backtrace:
    ▆
 1. └─traktok::tt_search_api(...)
 2.   └─traktok:::tt_query_request(...)
 3.     ├─httr2::resp_body_json(...)
 4.     │ └─httr2:::check_response(resp)
 5.     │   └─httr2:::is_response(resp)
 6.     └─httr2::req_perform(...)
 7.       └─httr2:::handle_resp(req, resp, error_call = error_call)
 8.         └─httr2:::resp_abort(resp, req, body, call = error_call)
 9.           └─rlang::abort(...)

The query is the same that I used before, except for the time span:

tt_search_api(query = query_elezioni,
                                         start_date = as.Date("2024-05-15"),
                                         end_date = as.Date("2024-05-21"),
                                         fields = "all",
                                         max_pages = 100, cache = T,
                                         verbose = T)

[fabiogiglietto]

I'm encountering the same @vot4ntonio error again. This time, the issue stems from pagination. The query executes correctly if it returns only one page, but it fails when multiple pages are returned.

A test with my custom code for the API confirmed that the issue is on the API side.

[fabiogiglietto]

@JBGruber @vot4ntonio, have either of you received any feedback from someone at TikTok regarding this? It appears they are transitioning to a Virtual Compute Environment, which I currently cannot access, and I'm unsure whom to contact for information

[JBGruber]

I'm in the unfortunate position that I changed institutions and can't access my mail anymore. I've opened a ticket about it and another one for this issue, but have never received a reply about either. And now I'm not sure I ever will ☹️

[vot4ntonio]

Unfortunately, I did not receive any response to my tickets.
However, I managed to collect the necessary data by trying every day until the situation was resolved.
My conclusion is this: there are some video IDs in a sort of limbo, indexed but not present in the data, which caused the API error. The same issue still occurs with tt_user_info: some missing users are handled by the package, while others I have to delete manually because they cause the procedure to fail.