To run ScrapydWeb in HTTPS mode

[ohenrik]

How do i enable https? I have a certificate file from let's encrypt and would like to start the server using https.

Is this possible at this time? It should be possible since using basic auth without HTTPS is dangerous.

[ohenrik]

Maybe adding config paths for the certificate files and optionally adding them here:
https://github.com/my8100/scrapydweb/blob/master/scrapydweb/run.py#L92

Would be enough?

You might also need to require pyopenssl (pip install pyopenssl)

app.run with ssl_context:

app.run(ssl_context=('cert.pem', 'key.pem'))

[my8100]

This works for me, and no need to install pyopenssl. Could you try and tell me your result?

context = ('/path/to/the/cert.pem', '/path/to/the/cert.key')
app.run(host=app.config['SCRAPYDWEB_BIND'], port=app.config['SCRAPYDWEB_PORT'], ssl_context=context)

[ohenrik]

It seems to work :)

[my8100]

What's the file type of your certificate? (Mine is '.pem')

[my8100]

Fixed in v1.1.0: Now you can set up ENABLE_HTTPS, CERTIFICATE_FILEPATH, and PRIVATEKEY_FILEPATH to enable HTTPS mode.

[jwjhdev]

Just wanted to point out to anyone having issues with SSL that scrapydweb uses assert os.path.isfile() to check the certificate and key files which only returns true if the path exists and is a file and you have (at least) read permissions.

Popular certificate aggregators like LetsEncrypt chmod the certificate as root only read/write.

That means you will need to run scrapydweb as root in order for the certificate to parse.

[my8100]

The below assert line could be commented out.

scrapydweb/scrapydweb/utils/check_app_config.py

Lines 84 to 91 in 7a3b81d

	check_assert('ENABLE_HTTPS', False, bool)
	if config.get('ENABLE_HTTPS', False):
	logger.info("HTTPS mode enabled: ENABLE_HTTPS = %s", config['ENABLE_HTTPS'])
	for k in ['CERTIFICATE_FILEPATH', 'PRIVATEKEY_FILEPATH']:
	check_assert(k, '', str, non_empty=True)
	assert os.path.isfile(config[k]), "%s not found: %s" % (k, config[k])
	logger.info("Running in HTTPS mode: %s, %s", config['CERTIFICATE_FILEPATH'], config['PRIVATEKEY_FILEPATH'])

[jwjhdev]

Thanks @my8100 👍

I can't think of a clean way to fix this issue other than acknowledging the file exists manually and commenting out that line as you say, would you agree?

• It should rightly fail without permission to read the cert/key, the assert os.path.isfile() is the correct method.
• Non-root users shouldn't change the permission of the cert/key as it is set to root as a layer of security
• Installing scrapyd, logparser and scrapydweb shouldn't be installed as root as a layer of security
• Running scrapydweb as a root user should also be avoided as a layer of security

Kind of leaves you in no man's land.

Thanks for a great tool by the way @my8100 and hello from Australia!

[my8100]

@jwjhdev
As the below comment said, HTTPS mode was not fully tested.
Thank you so much for the feedback.

scrapydweb/scrapydweb/default_settings.py

Lines 82 to 92 in 7a3b81d

	############################## ScrapydWeb #####################################
	# The default is False, set it to True and add both CERTIFICATE_FILEPATH and PRIVATEKEY_FILEPATH
	# to run ScrapydWeb in HTTPS mode.
	# Note that this feature is not fully tested, please leave your comment here if ScrapydWeb
	# raises any excepion at startup: https://github.com/my8100/scrapydweb/issues/18
	ENABLE_HTTPS = False
	# e.g. '/home/username/cert.pem'
	CERTIFICATE_FILEPATH = ''
	# e.g. '/home/username/cert.key'
	PRIVATEKEY_FILEPATH = ''