Build: allow to install packages with apt #8065

stsewd · 2021-03-31T23:51:30Z

All package names and forms like patters/version (package-*, package==2.x) are allowed
We don't allow injecting extra options nor install from a path (this is validated in the config file, you'll get a nice error with the exact package that is invalid)
Update and installation is done before the python setup and after the clone step

You can test this locally with this branch from the test-builds https://github.com/readthedocs/test-builds/blob/use-apt/.readthedocs.yaml

Implements #8060
Related: #7566

ericholscher

This looks great. I think we should look a bit more closely at the security aspects of it, but I think my changes will at least help:

$ sudo apt-get install --yes -- --quiet test
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package --quiet
E: Unable to locate package test

ericholscher · 2021-04-05T18:22:42Z

docs/config-file/v2.rst

@@ -64,7 +64,7 @@ This is to avoid typos and provide feedback on invalid configurations.

 .. contents::
   :local:
-   :depth: 1
+   :depth: 3


I wonder if this will spam the TOC too much -- seems fine tho?

I was between 2 or 3, but wanted to have build.apt_images and sphing.builder visible instead of just having sphinx and build, but no strong opinion.

docs/config-file/v2.rst

readthedocs/config/config.py

ericholscher · 2021-04-05T18:29:27Z

readthedocs/config/config.py

        return build

+    def validate_apt_package(self, index):


I feel like we probably can be smarter about this. Can we not call apt after a --, which is the shell way of handling the end of options?

Unless otherwise noted, each builtin command documented as accepting options preceded by ‘-’ accepts ‘--’ to signify the end of the options. The :, true, false, and test/[ builtins do not accept options and do not treat ‘--’ specially. The exit, logout, return, break, continue, let, and shift builtins accept and process arguments beginning with ‘-’ without requiring ‘--’. Other builtins that accept arguments but are not specified as accepting options interpret arguments beginning with ‘-’ as invalid options and require ‘--’ to prevent this interpretation.

We should still raise a validation error here, but I'd like to see additional restrictions for this built in.

Good point Eric.

Also, we need to take care of executing commands in the same line as well. Like,

apt-get install `touch /tmp/hello.txt`

I believe python's subprocess command will handle this, but we should definitely be sure and have tests for it.

Yes, subprocess should handle this as long as you don't pass shell=True. However, it's worth a test to make sure we don't ever mess that up.

I don't think we use subprocess, but the Docker client API (exec_create), tho.

We don't use subprocess, we escape special chars in

readthedocs.org/readthedocs/doc_builder/environments.py

Lines 335 to 367 in 2435a4a

def get_wrapped_command(self):

"""

Wrap command in a shell and optionally escape special bash characters.

In order to set the current working path inside a docker container, we

need to wrap the command in a shell call manually.

Some characters will be interpreted as shell characters without

escaping, such as: ``pip install requests<0.8``. When passing

``escape_command=True`` in the init method this escapes a good majority

of those characters.

"""

bash_escape_re = re.compile(

r"([\t\ \!\"\#\$\&\'\*\:\;\<\>\?\@"

r'\[\\\]\^\`\{\|\}\~])',

)

prefix = ''

if self.bin_path:

prefix += 'PATH={}:$PATH '.format(self.bin_path)

command = (

' '.join([

bash_escape_re.sub(r'\\\1', part) if self.escape_command else part

for part in self.command

])

)

return (

"/bin/sh -c 'cd {cwd} && {prefix}{cmd}'".format(

cwd=self.cwd,

prefix=prefix,

cmd=command,

)

)

Have you tested defining packages in a malicious way? (e.g. trying to executed forbidden things). From this docstring it only happen under certain circumstances. Are we sure we are always calling this in that way?

escape_command defaults to true if that's what you mean.

readthedocs/projects/tasks.py

ericholscher · 2021-04-05T18:31:51Z

readthedocs/projects/tasks.py

+                user=settings.RTD_BUILD_SUPER_USER,
+            )
+            self.build_env.run(
+                'apt-get', 'install', '-y', '-q', *packages,


We should always use the long-version of arguments in code, to make it clearer what it does. Also add in the -- Ending

Suggested change

'apt-get', 'install', '-y', '-q', *packages,

# -- ends all command arguments to protect against users passing them

'apt-get', 'install', '--assume-yes', '--quiet', '--', *packages,

Is there a reason we want this to be quiet? I know the output is sort of long but it might help folks debug their own problems. I tested it with --quiet and it's still fairly verbose but maybe a comment is appropriate on why.

I looked at what CircleCI does and they basically give users free reign to run arbitrary commands including apt-get install. Our system seems quite a bit more locked down than that which seems fine. There's still some risk (could a user install some docker utilities and somehow access the host?) but I think as long as we're just installing built-in Debian packages and sanitizing the inputs really well, we should be OK.

Is there a reason we want this to be quiet? I know the output is sort of long but it might help folks debug their own problems. I tested it with --quiet and it's still fairly verbose but maybe a comment is appropriate on why.

-q will suppress the progress bar not the output itself, -qq will suppress everything.

ericholscher · 2021-04-05T18:34:02Z

readthedocs/rtd_tests/tests/test_celery.py

+    @patch('readthedocs.projects.tasks.UpdateDocsTaskStep.setup_vcs', new=MagicMock)
+    @patch.object(BuildEnvironment, 'run')
+    @patch('readthedocs.doc_builder.config.load_config')
+    def test_install_apt_packages(self, load_config, run):


This seems like a lot of patches? Seems we're trying to stop the builds from processing, but I wonder if there's a cleaner way to do this?

I was able to remove one patch, but I think a cleaner way would require refactor the build task so we can mock self.build_env, self.setup_env and self.python_env

humitos

Looks good to me.

I also think we should take care a little deeper about security concerns here and make some extra QA for these cases.

docs/config-file/v2.rst

humitos · 2021-04-06T14:20:50Z

readthedocs/config/config.py

        return build

+    def validate_apt_package(self, index):


Good point Eric.

Also, we need to take care of executing commands in the same line as well. Like,

apt-get install `touch /tmp/hello.txt`

readthedocs/config/config.py

readthedocs/settings/base.py

ericholscher · 2021-04-06T15:52:47Z

Pinging @davidfischer here to chime in on the security angles of this. I think we've covered most of them (passing -- to end the argument parsing, tests for shelling out) but I feel like there's lots of terrible ways this can go wrong. In particular, we should write more about how the Python level protections work around client.exec_create in docker, and make sure we're thinking through all the various places this might break.

Co-authored-by: Eric Holscher <25510+ericholscher@users.noreply.github.com>

stsewd · 2021-04-06T19:52:30Z

readthedocs/config/config.py

+                '/',
+                '.',
+            ]
+            for start in invalid_starts:


I left this bc feels like it gives a nice error for usual mistakes, but isn't needed as the regex below already validates this.

readthedocs/config/config.py

ericholscher

The changes here look good. I think we can ship this soon 👍

ericholscher · 2021-05-03T18:00:13Z

docs/config-file/v2.rst

+.. note::
+
+   When possible avoid installing Python packages using apt (``python3-numpy`` for example),
+   :ref:`use pip or Conda instead <guides/reproducible-builds:pinning dependencies>`.


humitos

LGTM! 🎉 APT packages! 😻

docs/config-file/v2.rst

humitos · 2021-05-04T14:24:03Z

readthedocs/config/config.py

        return build

+    def validate_apt_package(self, index):


Have you tested defining packages in a malicious way? (e.g. trying to executed forbidden things). From this docstring it only happen under certain circumstances. Are we sure we are always calling this in that way?

jakirkham · 2021-05-12T18:20:02Z

I'm probably trying this too soon (sorry am really excited about this feature 😄). Is this the right way to use it ( rapidsai/ucx-py#734 )? Is there some way to tell the package is being installed? If I just need to wait a bit, happy to wait 🙂

stsewd · 2021-05-12T18:24:10Z

@jakirkham this change should be live by next tuesday in the afternoon.

stsewd · 2021-05-12T18:26:22Z

also, this will work only with the v2 of the conig file (you are using v1) https://docs.readthedocs.io/en/stable/config-file/v2.html

version: 2
build:
  apt_packages:
    - libnuma1

jakirkham · 2021-05-12T18:27:23Z

Awesome! Thank you for working on this 🙏

Sorry for getting ahead of things here 😅 Will give this another try next week 🙂

stsewd · 2021-05-18T16:57:43Z

@jakirkham and anyone else subscribed, this change is live now! Please let us know if you find any problems!

jakirkham · 2021-05-18T17:36:53Z

Worked great! 😄 Thanks again 🙏

stsewd force-pushed the install-apt-packages branch 7 times, most recently from 0608e29 to 4ca6676 Compare April 1, 2021 23:04

Build: allow to install packages with apt

2435a4a

stsewd force-pushed the install-apt-packages branch from 4ca6676 to 2435a4a Compare April 1, 2021 23:10

stsewd marked this pull request as ready for review April 1, 2021 23:26

stsewd requested a review from a team April 1, 2021 23:26

ericholscher reviewed Apr 5, 2021

View reviewed changes

jakirkham mentioned this pull request Apr 5, 2021

Build docs on CI rapidsai/ucx-py#278

Closed

humitos reviewed Apr 6, 2021

View reviewed changes

ericholscher requested a review from davidfischer April 6, 2021 15:50

stsewd and others added 3 commits April 6, 2021 12:43

Apply suggestions from code review

d355827

Co-authored-by: Eric Holscher <25510+ericholscher@users.noreply.github.com>

Updates from review

f6116fc

Merge branch 'master' into install-apt-packages

59768af

stsewd commented Apr 6, 2021

View reviewed changes

readthedocs/config/config.py Outdated Show resolved Hide resolved

stsewd added 3 commits April 6, 2021 15:01

Rename setting

45ab78f

Reduce mocks

b443949

Linter

e5923a1

stsewd requested review from ericholscher and humitos April 6, 2021 20:24

This was referenced Apr 17, 2021

Documentation is failing to build cekit/cekit#724

Closed

Restore ODCS to requirements.txt cekit/cekit#727

Closed

stsewd mentioned this pull request Apr 22, 2021

Replace current import process for Github with Actions integration #8133

Closed

Darylgolden mentioned this pull request Apr 26, 2021

OpenGL and rendered documentation ManimCommunity/manim#1214

Open

stsewd added 3 commits April 27, 2021 18:24

Merge branch 'master' into install-apt-packages

2773710

Keep it simple for now, don't allow versions/distributions

5f8874b

Update examples

ecd0183

charlesbluca mentioned this pull request May 3, 2021

UCX not building with NUMA support, impacting RTD builds rapidsai/ucx-py#728

Closed

ericholscher approved these changes May 3, 2021

View reviewed changes

humitos approved these changes May 4, 2021

View reviewed changes

This was referenced May 6, 2021

Feature request: provide libgdal/libgeos inside the build container #8160

Closed

Add libkrb5-dev to build gssapi readthedocs/readthedocs-docker-images#168

Closed

stsewd added 2 commits May 12, 2021 11:48

Merge branch 'master' into install-apt-packages

1d330ea

Simplify regex

f735a2d

stsewd enabled auto-merge (squash) May 12, 2021 17:51

stsewd merged commit 130b6b4 into master May 12, 2021

stsewd deleted the install-apt-packages branch May 12, 2021 17:58

jakirkham mentioned this pull request May 12, 2021

Install libnuma on RTD rapidsai/ucx-py#734

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build: allow to install packages with apt #8065

Build: allow to install packages with apt #8065

stsewd commented Mar 31, 2021 •

edited

Loading

ericholscher left a comment

ericholscher Apr 5, 2021

stsewd Apr 6, 2021

ericholscher Apr 5, 2021

humitos Apr 6, 2021

ericholscher Apr 6, 2021

davidfischer Apr 6, 2021

humitos Apr 6, 2021

stsewd Apr 6, 2021

humitos May 4, 2021

stsewd May 4, 2021

ericholscher Apr 5, 2021

davidfischer Apr 6, 2021

stsewd Apr 6, 2021

ericholscher Apr 5, 2021

stsewd Apr 6, 2021

humitos left a comment

humitos Apr 6, 2021

ericholscher commented Apr 6, 2021 •

edited

Loading

stsewd Apr 6, 2021

ericholscher left a comment

ericholscher May 3, 2021

humitos left a comment

humitos May 4, 2021

jakirkham commented May 12, 2021

stsewd commented May 12, 2021

stsewd commented May 12, 2021 •

edited

Loading

jakirkham commented May 12, 2021

stsewd commented May 18, 2021

jakirkham commented May 18, 2021

	def get_wrapped_command(self):
	"""
	Wrap command in a shell and optionally escape special bash characters.

	In order to set the current working path inside a docker container, we
	need to wrap the command in a shell call manually.

	Some characters will be interpreted as shell characters without
	escaping, such as: ``pip install requests<0.8``. When passing
	``escape_command=True`` in the init method this escapes a good majority
	of those characters.
	"""
	bash_escape_re = re.compile(
	r"([\t\ \!\"\#\$\&\'\(\)\*\:\;\<\>\?\@"
	r'\[\\\]\^\`\{\\|\}\~])',
	)
	prefix = ''
	if self.bin_path:
	prefix += 'PATH={}:$PATH '.format(self.bin_path)

	command = (
	' '.join([
	bash_escape_re.sub(r'\\\1', part) if self.escape_command else part
	for part in self.command
	])
	)
	return (
	"/bin/sh -c 'cd {cwd} && {prefix}{cmd}'".format(
	cwd=self.cwd,
	prefix=prefix,
	cmd=command,
	)
	)

	'apt-get', 'install', '-y', '-q', *packages,
	# -- ends all command arguments to protect against users passing them
	'apt-get', 'install', '--assume-yes', '--quiet', '--', *packages,

Build: allow to install packages with apt #8065

Build: allow to install packages with apt #8065

Conversation

stsewd commented Mar 31, 2021 • edited Loading

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ericholscher commented Apr 6, 2021 • edited Loading

Choose a reason for hiding this comment

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakirkham commented May 12, 2021

stsewd commented May 12, 2021

stsewd commented May 12, 2021 • edited Loading

jakirkham commented May 12, 2021

stsewd commented May 18, 2021

jakirkham commented May 18, 2021

stsewd commented Mar 31, 2021 •

edited

Loading

ericholscher commented Apr 6, 2021 •

edited

Loading

stsewd commented May 12, 2021 •

edited

Loading