Unable to retrive the full page of url

[peterpt]

I am using httpie 0.9.8 and when i try to download the webpage of dropbox i dont get the page fully downloaded , only partially .

Using this dropbox link :
https://www.dropbox.com/sh/erv1tycztizfvyd/AADeXwemV9sK37MSHqxmYz_5a?dl=0

and using the command :
http https://www.dropbox.com/sh/erv1tycztizfvyd/AADeXwemV9sK37MSHqxmYz_5a?dl=0 -o page.html

i get the links from 30 files where in reality exists 50 files inside that folder in dropbox if we opened with a normal web browser like firefox or chrome .

Any idea ?

$ http --debug <COMPLETE ARGUMENT LIST THAT TRIGGERS THE ERROR>
<COMPLETE OUTPUT>

Provide any additional information, screenshots, or code examples below:

[BoboTiG]

Could you try the latest HTTPie version (2.5.0 at the time)?

[peterpt]

test.zip

i have try it with 2.5.0 and still have the same issue .

I created a simple script that uses httpie and uses grep next to parse the html file and retrieve only the links , probably is more easier for you guys make tests , i am sending it as an attachment in zip file .

[Bhavye-Malhotra]

I have tested the script and it works perfectly fine. The issue seems fixed in version 2.6.0

[peterpt]

You have 50 links of wav files in that drop box , httpie only get 31 , your output was the same as i got in 2.5.0 .

[Bhavye-Malhotra]

Ohh, let me recheck and try to fix it.

[Bhavye-Malhotra]

@peterpt hey, I tried to extract the URLs manually by downloading the source HTML page locally and running the script on that HTML file and I found out that it still shows 31 links and not 48 which is the total number of files so I think its a thing from dropbox side and not httpie 😅
hope it helps.

[isidentical]

The reason there are only 30-something in the HTTPie usage and 50-something on the browser is that once the page is fully loaded, the browser starts executing javascript which does light pagination to load more entries. If you want to test this out by yourself, simply block the https://www.dropbox.com/list_shared_link_folder_entries API on your browser's command toolkit and then try again. This time, you'll also see only 30-something entries.

I am afraid this is something httpie does not support but it can be achieved by other means (e.g using a headless browser via selenium to fully load the page and then do the extraction).