private apt env (SC-1346) #2342

renanrodrigo · 2022-12-04T18:00:16Z

Stop relying on the unauthenticated and pinned-to-never ESM repositories in the apt config and use a private separate apt cache for that instead.

Reference is spec US028.

LP: #1990378

todos:

remove postinst code that would create files in the system apt configuration
remove timer job that would do the same
Implement functionality to create the internal cache
create/update internal cache from the apt update hook
make sure the internal cache and system cache are fine when enabling/disabling esm-[infra/apps]
change security status to read from both caches
change the C++ hook to read from both caches
create/update internal cache when running dist-upgrade (decided to remove it and let be recreated on update)
make sure all integration tests pass and make sense

Desired commit type

Squash and merge: Using the "Proposed Commit Message"
Create a merge commit and use "Proposed Commit Message"
Rebase and merge, no merge commit, rely only on existing commit messages

Checklist

I have updated or added any unit tests accordingly
I have updated or added any integration tests accordingly
I have updated or added any documentation accordingly

Does this PR require extra reviews?

Yes: Julian Klode, Robie Basak (or other SRU member for pre-review)
No

uaclient/apt.py

uaclient/entitlements/esm.py

debian/ubuntu-advantage-tools.postinst

uaclient/security_status.py

orndorffgrant · 2022-12-16T21:49:25Z

cpp hook portion of this is mostly done and sitting at #2358 for now. we can decide how to rebase/merge all of this together in the new year

renanrodrigo · 2023-01-02T22:15:38Z

Rebased this branch on main, then addressed the review comments here and did small changes for things to work as expected

renanrodrigo · 2023-01-03T11:47:36Z

Imported the commit from @orndorffgrant's branch here for completeness

uaclient/apt.py

uaclient/entitlements/esm.py

uaclient/jobs/update_messaging.py

uaclient/apt.py

julian-klode

I've only reviewed the apt related bits and not very deeply but I did not see anything wrong there.

basak

This is not a full review. I've taken the code at face value and have assumed that it does what it's intended to do and your testing is adequate.

My comments are all optional - I'd like you to consider them as suggestions for improvement but nothing is a hard blocker. Some of them relate to style that wouldn't be appropriate to fix in this PR anyway.

I appreciate how nicely this is laid out in separate commits. This made it straightforward to review. Thanks!

uaclient/security_status.py

lib/upgrade_lts_contract.py

debian/ubuntu-advantage-tools.postinst

uaclient/apt.py

uaclient/entitlements/esm.py

uaclient/apt.py

uaclient/entitlements/esm.py

apt-hook/20apt-esm-hook.conf

systemd/esm-cache.service

apt-hook/json-hook.cc

Remove all the logic to setup unauthenticated repos LP: #1990378 Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

Other logic will be used to ensure the esm packages will be advertised Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

This service is executed as part of the pre-invoke apt hook. Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

Remove all enable/disable/logic status relying on it Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

When the user does not have esm enabled, we will need the security_status module to read esm package information from the private cache. We are updating the code to support that path

When running the apt dist-upgrade command, we want to remove the private esm apt cache, since it will be targeting the old release. When the user runs apt update again, the private apt cache will be created again for the right release Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

renanrodrigo marked this pull request as draft December 4, 2022 18:04

renanrodrigo force-pushed the private-apt branch from 46fcc60 to a631888 Compare December 7, 2022 04:56

renanrodrigo changed the title ~~wip: private apt env~~ wip: private apt env (SC-1346) Dec 7, 2022

lucasmoura reviewed Dec 13, 2022

View reviewed changes

uaclient/apt.py Outdated Show resolved Hide resolved

uaclient/entitlements/esm.py Outdated Show resolved Hide resolved

debian/ubuntu-advantage-tools.postinst Show resolved Hide resolved

renanrodrigo commented Dec 15, 2022

View reviewed changes

uaclient/security_status.py Outdated Show resolved Hide resolved

uaclient/security_status.py Outdated Show resolved Hide resolved

renanrodrigo force-pushed the private-apt branch from 81e9230 to 45e3bf9 Compare January 2, 2023 22:14

renanrodrigo force-pushed the private-apt branch from 45e3bf9 to b5f4116 Compare January 3, 2023 11:33

lucasmoura reviewed Jan 3, 2023

View reviewed changes

uaclient/apt.py Show resolved Hide resolved

uaclient/apt.py Outdated Show resolved Hide resolved

uaclient/entitlements/esm.py Show resolved Hide resolved

uaclient/jobs/update_messaging.py Show resolved Hide resolved

orndorffgrant reviewed Jan 3, 2023

View reviewed changes

uaclient/apt.py Outdated Show resolved Hide resolved

orndorffgrant mentioned this pull request Jan 3, 2023

wip: cpp hook using esm apt cache #2358

Closed

8 tasks

renanrodrigo force-pushed the private-apt branch from dfaa16d to baa8895 Compare January 4, 2023 15:13

renanrodrigo marked this pull request as ready for review January 4, 2023 20:25

renanrodrigo force-pushed the private-apt branch 3 times, most recently from aef2c4c to 74c2eff Compare January 5, 2023 16:57

renanrodrigo requested review from julian-klode and basak January 5, 2023 17:02

renanrodrigo force-pushed the private-apt branch 2 times, most recently from 40cea43 to becd0ac Compare January 6, 2023 13:30

orndorffgrant changed the title ~~wip: private apt env (SC-1346)~~ private apt env (SC-1346) Jan 6, 2023

renanrodrigo force-pushed the private-apt branch 2 times, most recently from d713171 to 172a6e3 Compare January 9, 2023 07:20

julian-klode approved these changes Jan 11, 2023

View reviewed changes

basak approved these changes Jan 12, 2023

View reviewed changes

renanrodrigo force-pushed the private-apt branch 2 times, most recently from 2ef82d9 to fccf893 Compare January 13, 2023 03:30

d/postinst: remove any unauthenticated esm repo from the Xenial systems

4e8b7e1

Remove all the logic to setup unauthenticated repos LP: #1990378 Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

renanrodrigo and others added 10 commits January 13, 2023 21:36

jobs: remove unused job to check for EOL

1ee8ad8

Other logic will be used to ensure the esm packages will be advertised Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

apt: add functionality to create and update the local esm cache

73033d4

Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

services: add new service to update local ESM caches

4479199

This service is executed as part of the pre-invoke apt hook. Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

esm: never pin repositories to 'never'

6faf1b5

Remove all enable/disable/logic status relying on it Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

esm: update local repos on enable and disable

7dff509

Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

security-status: use private cache when esm is disabled

1bf40d7

When the user does not have esm enabled, we will need the security_status module to read esm package information from the private cache. We are updating the code to support that path

apt-hook: update cpp hook to use the esm apt cache

b1e3ab8

chore: rename 'remove' functions to clarify idempotency

983bb06

Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

chore: import apt explicitly for readability

a304d63

Signed-off-by: Renan Rodrigo <renanrodrigo@canonical.com>

renanrodrigo force-pushed the private-apt branch from fccf893 to a304d63 Compare January 14, 2023 00:46

renanrodrigo merged commit b4c3778 into main Jan 14, 2023

renanrodrigo deleted the private-apt branch January 14, 2023 01:58

renanrodrigo mentioned this pull request Jan 8, 2024

Convert repo files to deb822 #2907

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

private apt env (SC-1346) #2342

private apt env (SC-1346) #2342

renanrodrigo commented Dec 4, 2022 •

edited

orndorffgrant commented Dec 16, 2022

renanrodrigo commented Jan 2, 2023

renanrodrigo commented Jan 3, 2023

julian-klode left a comment

basak left a comment

private apt env (SC-1346) #2342

private apt env (SC-1346) #2342

Conversation

renanrodrigo commented Dec 4, 2022 • edited

Desired commit type

Checklist

Does this PR require extra reviews?

orndorffgrant commented Dec 16, 2022

renanrodrigo commented Jan 2, 2023

renanrodrigo commented Jan 3, 2023

julian-klode left a comment

Choose a reason for hiding this comment

basak left a comment

Choose a reason for hiding this comment

renanrodrigo commented Dec 4, 2022 •

edited