Manage `sources.list` and make bootstrap script purge lists before updating

[edunham]

Ran into SystemError: E:Unable to parse package file /var/lib/apt/lists/us-west-2.ec2.archive.ubuntu.com_ubuntu_dists_trusty-updates_main_i18n_Translation-en (1), E:The package lists or status file could not be parsed or opened. again today. We've been running into similar apt issues every week or two on the linux slaves.

@aneeshusa, @larsbergstrom, what are your thoughts on adding a salt state to rm -r /var/lib/apt/lists/; sudo apt-get update before applying other states on the Linux hosts, to circumvent future instances of this issue?

[aneeshusa]

I did some searching and this appears to be a problem with EC2's apt mirroring. In the interest of being more reproducible, here is what I think is a better solution:

• Use Salt to manage the /etc/apt/sources.list file, like we already manage files in /etc/apt/sources.list.d/, so that it never points at EC2 mirrors.
• Potentially update the bootstrap script to replace the sources.list file and run rm -r /var/lib/apt/lists/* and sudo apt-get update to make sure new machines don't exhibit this problem.
• Manually run rm -r /var/lib/apt/lists/* and sudo apt-get update on the existing machines.

[edunham]

Sounds good to me. This would be an appropriate first bug for someone with intermediate Ubuntu familiarity :)

[choudhary001]

I would like give this a go. Where should i start with this?

[aneeshusa]

@choudhary001 Great, it's yours :)

Are you familiar with apts /etc/apt/sources.list file? A good starting point would be adding a new file.managed state to the ubuntu/init.sls file which locks down the contents of the /etc/apt/sources.list file on our Ubuntu builders, using official Ubuntu mirrors instead of EC2 mirrors.

[choudhary001]

@aneeshusa I am familiar with /etc/apt/sources.list file. I looked into 'https://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html' as well. But I didn't understand the locking down contents of the sources.list file, using official Ubuntu mirrors instead of EC2 mirrors part. Could explain it a bit more?

[aneeshusa]

@choudhary001 Salt, among other things, is a configuration management tool. The file.managed state lets us specify the exact contents a file should have, so I just want to grab the sources.list from a regular Ubuntu install (which points at the official Ubuntu mirrors) and use file.managed to set that as the contents for /etc/apt/sources.list on our infra.

This will entail checking in a good copy of sources.list, and pointing the source parameter of file.managed to that checked-in copy.

[jdm]

@choudhary001 Are you still working on this?

[choudhary001]

@jdm Yes

[KiChjang]

@choudhary001 Are you stuck on anything? Can we hear what is going on right now?

[choudhary001]

@KiChjang I made the changes suggested by @aneeshusa as follow:
created a sources.list file in ~/ubuntu/apt/sources.list using the contents of /etc/apt/sources.list from servo-master1 vm and modified ~/ubuntu/init.sls as below:

--- a/ubuntu/init.sls
+++ b/ubuntu/init.sls
@@ -23,3 +23,11 @@
- group
- mode
- clean: True
+
+/etc/apt/sources.list:

• file.managed:
• • user: root
• • group: root
• • mode: 755
• • source: salt://{{ tpldir }}/apt/sources.list

After this I created vm images again but I don't see the source.list file being picked from repository. I was looking for was to test this. Basically stuck on how to verify this works. Any pointers will help.

Sorry that it got delayed this much.

[aneeshusa]

@choudhary001, it will be easier for me to help if you open a WIP PR with what you have so far. It's hard to tell but there may be a path issue or something else.

One way to test this is manual verification inside a Vagrant VM, and another would be adding a test that makes sure that the deployed sources.list file doesn't mention any EC2 mirrors (easiest way is probably searching for some strings).

[choudhary001]

Hi @aneeshusa ,
I have created this pull request - #520. Could you please take a look and help me further.

[choudhary001]

@aneeshusa
Hi,
You proposed below as the solution for this problem:

1. Use Salt to manage the /etc/apt/sources.list file, like we already manage files in /etc/apt/sources.list.d/, so that it never points at EC2 mirrors.
2. Potentially update the bootstrap script to replace the sources.list file and run rm -r /var/lib/apt/lists/* and sudo apt-get update to make sure new machines don't exhibit this problem.
3. Manually run rm -r /var/lib/apt/lists/* and sudo apt-get update on the existing machines.

I think no. 1 is done. Do I need to do anything for no. 2 and no. 3 here?
Thanks!

[jdm]

@aneeshusa Could you answer the questions in the previous comment?

[aneeshusa]

#520 takes care of the first item (although it currently fails on Travis and I'm not sure why).

The second item needs the wiki to be updated after the PR lands, and the third item will be manually done as part of deploy; you don't need to do anything for those.

If you can get the PR tests working on Travis, then we'll handle everything else.

[jdhorwitz]

If there is still work to be done on this and someone is willing to mentor me along the way I would love to work on it.

[choudhary001]

@aneeshusa @jdm Hi, I have submitted a patch to fix issues due to #582 in my on-going pull request - #520.
I apologize for the late response, I was busy in the previous months.

[choudhary001]

@jdm @aneeshusa did you get a chance look into new changes in #582

[jdm]

Do you mean #520?

[choudhary001]

Yes, I meant #520. Sorry for the link mess up.