"Too many redirects" error should not be retried

[alexey-milovidov]

Use case

milovidov-desktop :) SELECT * FROM url('https://google.com/', LineAsString)

SELECT *
FROM url('https://google.com/', LineAsString)

Query id: a8004dfc-3223-4cef-a8ac-c3b9d8b84ada

2024.01.19 03:14:41.378567 [ 31120 ] {} <Trace> LOCAL-Session: 2c8e4d9a-076c-4112-8e36-352f2a7d9864 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2024.01.19 03:14:41.378729 [ 31120 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Debug> executeQuery: (from 0.0.0.0:0, user: ) SELECT * FROM url('https://google.com/', LineAsString) (stage: Complete)
2024.01.19 03:14:41.378858 [ 31120 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ContextAccess (default): Access granted: CREATE TEMPORARY TABLE, URL ON *.*
2024.01.19 03:14:41.379194 [ 31120 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2024.01.19 03:14:41.379625 [ 31131 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
2024.01.19 03:14:41.412476 [ 31131 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> HTTPSessionAdapter: Created HTTP(S) session with google.com:443 (2a00:1450:400e:811::200e:443)
2024.01.19 03:14:41.433004 [ 31131 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: Failed to make HTTP_HEAD request to https://google.com/. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.
2024.01.19 03:14:41.433039 [ 31131 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://www.google.com/
2024.01.19 03:14:41.467470 [ 31131 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> HTTPSessionAdapter: Created HTTP(S) session with google.com:443 (2a00:1450:400e:811::200e:443)
↑ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:41.488333 [ 31131 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> FormatFactory: Failed to setup ParallelReadBuffer because of an exception:
DB::Exception: Cannot find out file size for: https://google.com/.
Falling back to the single-threaded buffer
2024.01.19 03:14:41.488375 [ 31131 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ParallelParsingInputFormat: Parallel parsing is used
2024.01.19 03:14:41.488619 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
2024.01.19 03:14:41.523857 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> HTTPSessionAdapter: Created HTTP(S) session with google.com:443 (2a00:1450:400e:811::200e:443)
2024.01.19 03:14:41.548199 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://google.com/` failed at try 1/10 with bytes read: 0/unknown. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.. (Current backoff wait is 100/10000 ms)
↗ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:41.648354 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
→ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:41.704685 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://google.com/` failed at try 2/10 with bytes read: 0/unknown. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.. (Current backoff wait is 200/10000 ms)
↓ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:41.904843 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
2024.01.19 03:14:41.966351 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://google.com/` failed at try 3/10 with bytes read: 0/unknown. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.. (Current backoff wait is 400/10000 ms)
↑ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:42.366521 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
↗ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:42.412692 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://google.com/` failed at try 4/10 with bytes read: 0/unknown. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.. (Current backoff wait is 800/10000 ms)
↗ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:43.212864 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
2024.01.19 03:14:43.266858 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://google.com/` failed at try 5/10 with bytes read: 0/unknown. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.. (Current backoff wait is 1600/10000 ms)
↗ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:44.867035 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
→ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:44.929190 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://google.com/` failed at try 6/10 with bytes read: 0/unknown. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.. (Current backoff wait is 3200/10000 ms)
→ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)                                                                                                                                                                                                                                          2024.01.19 03:14:48.129391 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Trace> ReadWriteBufferFromHTTP: Sending request to https://google.com/
2024.01.19 03:14:48.184954 [ 31125 ] {a8004dfc-3223-4cef-a8ac-c3b9d8b84ada} <Error> ReadWriteBufferFromHTTP: HTTP request to `https://google.com/` failed at try 7/10 with bytes read: 0/unknown. Error: DB::Exception: Too many redirects while trying to access https://google.com/. You can allow redirects by changing the setting 'max_http_get_redirects'. Example: `SET max_http_get_redirects = 10`. Redirects are restricted to prevent possible attack when a malicious server redirects to an internal resource, bypassing the authentication or firewall.. (Current backoff wait is 6400/10000 ms)

[skyoct]

@alexey-milovidov Hi, I want to give it a try

[alexey-milovidov]

@skyoct, thank you! This will be very helpful.