Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specifying User Agent: required & appropriate string syntax #42

Closed
howardm opened this issue Aug 25, 2019 · 7 comments
Closed

Specifying User Agent: required & appropriate string syntax #42

howardm opened this issue Aug 25, 2019 · 7 comments
Labels
bug Bugs and defects (faults of monolith, not target websites)

Comments

@howardm
Copy link

howardm commented Aug 25, 2019

Hi,

I tried monlith with the following:

monolith https://www.nytimes.com/2019/08/25/world/europe/g7-iran-trump-biarritz.html > test.html

I got the following error message:

error: The following required arguments were not provided: --user-agent <Iceweasel>

I thought this was optional.

If not, what is an example of the proper syntax for providing this argument ?

Thanks,

Howard

@snshn snshn added the bug Bugs and defects (faults of monolith, not target websites) label Aug 25, 2019
@snshn
Copy link
Member

snshn commented Aug 25, 2019

Hi Howard,

it might've been a bug that slipped in one of the earlier versions. Could you please git pull the latest version and try it again?

@howardm
Copy link
Author

howardm commented Aug 25, 2019

Hi,

O.K. I just did that.

I still get the same error message (as above).

Howard

@snshn
Copy link
Member

snshn commented Aug 25, 2019

You may need to run cargo install --path . -f to build the latest version and then overwrite the previous local installation of monolith. I was able to save the page you mentioned above using the latest version, here's my output:

$ monolith -V
monolith 2.0.14
$ monolith https://www.nytimes.com/2019/08/25/world/europe/g7-iran-trump-biarritz.html > test.html
[ https://www.nytimes.com/2019/08/25/world/europe/g7-iran-trump-biarritz.html ]
[ https://www.nytimes.com/vi-assets/static-assets/favicon-4bf96cb6a1093748bf5b3c429accb9b4.ico ]
[ https://www.nytimes.com/vi-assets/static-assets/apple-touch-icon-319373aaf4524d94d38aa599c56b8655.png ]
[ https://www.nytimes.com/vi-assets/static-assets/global-42db6c8821fec0e2b3837b2ea2ece8fe.css ]
[ https://static01.nyt.com/images/2019/08/25/world/25G7-iran-1/merlin_159699912_13079c6d-a173-4b31-9a46-efefd28f9360-articleLarge.jpg?quality=75&auto=webp&disable=upscale ]
[ https://static01.nyt.com/images/2019/08/25/world/25G7-iran-2/merlin_159700404_88347fa3-6716-4ae0-a567-8bebece14794-articleLarge.jpg?quality=75&auto=webp&disable=upscale ]
[ https://www.nytimes.com/vi-assets/static-assets/adslot-c6ffe1b749add526a06c.js ]
[ https://www.nytimes.com/vi-assets/static-assets/vendor-3389f9c978bdc7cb443c.js ]
[ https://www.nytimes.com/vi-assets/static-assets/story-b70297033149b8af8853.js ]
[ https://www.nytimes.com/vi-assets/static-assets/main-e2538404741407daf3c0.js ]
[ https://www.googletagmanager.com/gtm.js?id=GTM-P528B3&gtm_auth=tfAzqo1rYDLgYhmTnSjPqw&gtm_preview=env-130&gtm_cookies_win=x ]

@snshn
Copy link
Member

snshn commented Aug 25, 2019

I would recommend using the -j flag before the URL to save the page without scripts, as it seems like NYT attempts to display some modal/overlay upon page load (to sell some paid subscription or something like that).

@howardm
Copy link
Author

howardm commented Aug 25, 2019 via email

@snshn
Copy link
Member

snshn commented Aug 25, 2019

Sure! To update to the latest version this should do the trick:

cd monolith
git pull
cargo install --force --path .

After doing that, this should save the page and render it as a readable document while offline:

monolith -j https://www.nytimes.com/2019/08/25/world/europe/g7-iran-trump-biarritz.html > test.html

Please let me know if it worked for you.

@howardm
Copy link
Author

howardm commented Aug 25, 2019 via email

@snshn snshn closed this as completed Aug 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bugs and defects (faults of monolith, not target websites)
Projects
None yet
Development

No branches or pull requests

2 participants