PyParted and a large rewrite of the underlying partitioning (#1604)

* Invert mypy files * Add optional pre-commit hooks * New profile structure * Serialize profiles * Use profile instead of classmethod * Custom profile setup * Separator between back * Support profile import via url * Move profiles module * Refactor files * Remove symlink * Add user to docker group * Update schema description * Handle list services * mypy fixes * mypy fixes * Rename profilesv2 to profiles * flake8 * mypy again * Support selecting DM * Fix mypy * Cleanup * Update greeter setting * Update schema * Revert toml changes * Poc external dependencies * Dependency support * New encryption menu * flake8 * Mypy and flake8 * Unify lsblk command * Update bootloader configuration * Git hooks * Fix import * Pyparted * Remove custom font setting * flake8 * Remove default preview * Manual partitioning menu * Update structure * Disk configuration * Update filesystem * luks2 encryption * Everything works until installation * Btrfsutil * Btrfs handling * Update btrfs * Save encryption config * Fix pipewire issue * Update mypy version * Update all pre-commit * Update package versions * Revert audio/pipewire * Merge master PRs * Add master changes * Merge master changes * Small renaming * Pull master changes * Reset disk enc after disk config change * Generate locals * Update naming * Fix imports * Fix broken sync * Fix pre selection on table menu * Profile menu * Update profile * Fix post_install * Added python-pyparted to PKGBUILD, this requires [testing] to be enabled in order to run makepkg. Package still works via python -m build etc. * Swaped around some setuptools logic in pyproject Since we define `package-data` and `packages` there should be no need for: ``` [tool.setuptools.packages.find] where = ["archinstall", "archinstall.*"] ``` * Removed pyproject collisions. Duplicate definitions. * Made sure pyproject.toml includes languages * Add example and update README * Fix pyproject issues * Generate locale * Refactor imports * Simplify imports * Add profile description and package examples * Align code * Fix mypy * Simplify imports * Fix saving config * Fix wrong luks merge * Refactor installation * Fix cdrom device loading * Fix wrongly merged code * Fix imports and greeter * Don't terminate on partprobe error * Use specific path on partprobe from luks * Update archinstall/lib/disk/device_model.py Co-authored-by: codefiles <11915375+codefiles@users.noreply.github.com> * Update archinstall/lib/disk/device_model.py Co-authored-by: codefiles <11915375+codefiles@users.noreply.github.com> * Update github workflow to test archinstall installation * Update sway merge * Generate locales * Update workflow --------- Co-authored-by: Daniel Girtler <girtler.daniel@gmail.com> Co-authored-by: Anton Hvornum <anton@hvornum.se> Co-authored-by: Anton Hvornum <anton.feeds+github@gmail.com> Co-authored-by: codefiles <11915375+codefiles@users.noreply.github.com>
archlinux · Apr 19, 2023 · 00b0ae7 · 00b0ae7
1 parent 5253e57
commit 00b0ae7
Show file tree

Hide file tree

Showing 201 changed files with 13,474 additions and 10,357 deletions.
diff --git a/.github/workflows/iso-build.yaml b/.github/workflows/iso-build.yaml
@@ -32,17 +32,7 @@ jobs:
       - run: cat /etc/os-release
       - run: pacman-key --init
       - run: pacman --noconfirm -Sy archlinux-keyring
-      - run: mkdir -p /tmp/archlive/airootfs/root/archinstall-git; cp -r . /tmp/archlive/airootfs/root/archinstall-git
-      - run: echo "pip uninstall archinstall -y; cd archinstall-git; rm -rf dist; python -m build -n; pip install dist/archinstall*.whl" > /tmp/archlive/airootfs/root/.zprofile
-      - run: echo "echo \"This is an unofficial ISO for development and testing of archinstall. No support will be provided.\"" >> /tmp/archlive/airootfs/root/.zprofile
-      - run: echo "echo \"This ISO was built from Git SHA $GITHUB_SHA\"" >> /tmp/archlive/airootfs/root/.zprofile
-      - run: echo "echo \"Type archinstall to launch the installer.\"" >> /tmp/archlive/airootfs/root/.zprofile
-      - run: cat /tmp/archlive/airootfs/root/.zprofile
-      - run: pacman -Sy; pacman --noconfirm -S git archiso
-      - run: cp -r /usr/share/archiso/configs/releng/* /tmp/archlive
-      - run: echo -e "git\npython\npython-pip\npython-build\npython-flit\npython-setuptools\npython-wheel" >> /tmp/archlive/packages.x86_64
-      - run: find /tmp/archlive
-      - run: cd /tmp/archlive;  mkarchiso -v -w work/ -o out/ ./
+      - run: ./build_iso.sh
       - uses: actions/upload-artifact@v3
         with:
           name: Arch Live ISO

diff --git a/.github/workflows/mypy.yaml b/.github/workflows/mypy.yaml
@@ -15,8 +15,4 @@ jobs:
             # one day this will be enabled
               # run: mypy --strict --module archinstall || exit 0
             - name: run mypy
-              run: mypy --follow-imports=silent archinstall/lib/menu/abstract_menu.py archinstall/lib/menu/global_menu.py
-                archinstall/lib/models/network_configuration.py archinstall/lib/menu/list_manager.py archinstall/lib/user_interaction/network_conf.py archinstall/lib/models/users.py
-                archinstall/lib/disk/blockdevice.py archinstall/lib/user_interaction/subvolume_config.py archinstall/lib/disk/btrfs/btrfs_helpers.py
-                archinstall/lib/translationhandler.py archinstall/lib/disk/diskinfo.py archinstall/lib/menu/table_selection_menu.py archinstall/lib/hsm
-                archinstall/lib/disk/encryption.py archinstall/lib/models/disk_encryption.py
+              run: mypy --config-file mypy.ini
diff --git a/.github/workflows/python-build.yml b/.github/workflows/python-build.yml
@@ -7,20 +7,37 @@ on: [ push, pull_request ]
 jobs:
   deploy:
     runs-on: ubuntu-latest
+    container:
+      image: archlinux:latest
+      options: --privileged
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python
         uses: actions/setup-python@v4
         with:
-          python-version: '3.x'
-      - name: Install dependencies
+          python-version: '3.10'
+      - name: Prepare arch
+        run: |
+          pacman-key --init
+          pacman --noconfirm -Sy archlinux-keyring
+          pacman --noconfirm -Sy python-pyparted pkgconfig gcc
+      - name: Install build dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install build twine
+          pip install --upgrade build twine wheel setuptools installer
+          pip uninstall archinstall -y
+      - name: Install package dependencies
+        run: |
+          pip install --upgrade simple-term-menu pyparted
       - name: Build archinstall
+        run: python -m build --wheel --no-isolation
+      - name: Install archinstall
+        run: python -m installer dist/*.whl
+      - name: Run archinstall
         run: |
-          python -m build . --wheel
+          python -V
+          archinstall -v
       - uses: actions/upload-artifact@v3
         with:
           name: archinstall
-          path: dist/*
+          path: dist/*
diff --git a/.gitignore b/.gitignore
@@ -33,4 +33,5 @@ venv
 **/cmd_history.txt
 **/*.*~
 /*.sig
-/*.json
+/*.json
+requirements.txt
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,42 @@
+default_stages: ['commit']
+repos:
+  - repo: https://github.com/pycqa/autoflake
+    rev: v2.0.2
+    hooks:
+      - id: autoflake
+        args: [
+          '--in-place',
+          '--remove-all-unused-imports',
+          '--ignore-init-module-imports'
+        ]
+        files: \.py$
+        require_serial: true
+        fail_fast: true
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
+    hooks:
+    # general hooks:
+    - id: check-added-large-files  # Prevent giant files from being committed
+      args: ['--maxkb=5000']
+    - id: check-merge-conflict  # Check for files that contain merge conflict strings
+    - id: check-symlinks  # Checks for symlinks which do not point to anything
+    - id: check-yaml  # Attempts to load all yaml files to verify syntax
+    - id: destroyed-symlinks  # Detects symlinks which are changed to regular files
+    - id: detect-private-key  # Checks for the existence of private keys
+    - id: end-of-file-fixer  # Makes sure files end in a newline and only a newline
+    - id: trailing-whitespace  # Trims trailing whitespace
+    # Python specific hooks:
+    - id: check-ast  # Simply check whether files parse as valid python
+    - id: check-docstring-first  # Checks for a common error of placing code before the docstring
+  - repo: https://github.com/pycqa/flake8
+    rev: 6.0.0
+    hooks:
+    - id: flake8
+      args: [--config=.flake8]
+      fail_fast: true
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.1.1
+    hooks:
+    - id: mypy
+      args: [--config=mypy.ini]
+      fail_fast: true
diff --git a/PKGBUILD b/PKGBUILD
@@ -13,6 +13,7 @@ license=(GPL3)
 depends=(
   'python'
   'systemd'
+  'python-pyparted'
 )
 makedepends=(
   'python-setuptools'
@@ -48,7 +49,6 @@ prepare() {
 
   # use real directories for examples and profiles, as symlinks do not work
   rm -fv $pkgname/{examples,profiles}
-  mv -v examples profiles $pkgname/
 }
 
 build() {

diff --git a/README.md b/README.md
@@ -26,44 +26,60 @@ Assuming you are on an Arch Linux live-ISO:
 
     # archinstall
 
+#### Advanced
 Some additional options that are not needed by most users are hidden behind the `--advanced` flag.
 
 ## Running from a declarative configuration file or URL
 
-Prerequisites:
-   1. Edit the [configuration file](https://github.com/archlinux/archinstall/blob/master/examples/config-sample.json) according to your requirements.
+`archinstall` can be run with a JSON configuration file. There are 2 different configuration files to consider,
+the `user_configuration.json` contains all general installation configuration, whereas the `user_credentials.json`
+contains the sensitive user configuration such as user password, root password and encryption password.
 
-Assuming you are on a Arch Linux live-ISO and booted into EFI mode.
+An example of the user configuration file can be found here
+[configuration file](https://github.com/archlinux/archinstall/blob/master/examples/config-sample.json)
+and example of the credentials configuration here
+[credentials file](https://github.com/archlinux/archinstall/blob/master/examples/creds-sample.json).
 
-    # archinstall --config <path to user config file or URL> --disk-layout <path to disk layout config file or URL> --creds <path to user credentials config file or URL>
+**HINT:** The configuration files can be auto-generated by starting `archisntall`, configuring all desired menu
+points and then going to `Save configuration`.
+
+To load the configuration file into `archinstall` run the following command
+```
+archinstall --config <path to user config file or URL> --creds <path to user credentials config file or URL>
+```
 
 # Available Languages
 
-Archinstall is available in different languages which have been contributed and are maintained by the community.  
+Archinstall is available in different languages which have been contributed and are maintained by the community.
 Current translations are listed below and vary in the amount of translations per language
 ```
 English
-Deutsch
-Español
-Français
+Arabic
+Brazilian Portuguese
+Czech
+Dutch
+French
+Georgian
+German
 Indonesian
-Italiano
-Nederlands
-Polskie
-Português do Brasil
-Português
-Svenska
-Türkçe
-čeština
-Русский
-اردو
-Ελληνικά
-தமிழ்
+Italian
+Korean
+Modern Greek
+Polish
+Portuguese
+Russian
+Spanish
+Swedish
+Tamil
+Turkish
+Ukrainian
+Urdu
 ```
 
-Any contributions to the translations are more than welcome, and to get started please follow [the guide](https://github.com/archlinux/archinstall/blob/master/archinstall/locales/README.md)
+Any contributions to the translations are more than welcome,
+to get started please follow [the guide](https://github.com/archlinux/archinstall/blob/master/archinstall/locales/README.md)
 
-# Help?
+# Help or Issues
 
 Submit an issue here on GitHub, or submit a post in the discord help channel.<br>
 When doing so, attach the `/var/log/archinstall/install.log` to the issue ticket. This helps us help you!
@@ -86,73 +102,111 @@ Therefore, Archinstall will try its best to not introduce any breaking changes e
 
 # Scripting your own installation
 
-You could just copy [guided.py](https://github.com/archlinux/archinstall/blob/master/examples/guided.py) as a starting point.
-
-However, assuming you're building your own ISO and want to create an automated installation process, or you want to install virtual machines onto local disk images, here is a [minimal example](https://github.com/archlinux/archinstall/blob/master/examples/minimal.py) of how to install using archinstall as a Python library:<br>
-
-```python
-import archinstall, getpass
-
-# Select a harddrive and a disk password
-harddrive = archinstall.select_disk(archinstall.all_blockdevices(partitions=False))
-disk_password = getpass.getpass(prompt='Disk password (won\'t echo): ')
-
-# We disable safety precautions in the library that protects the partitions
-harddrive.keep_partitions = False
-
-# First, we configure the basic filesystem layout
-with archinstall.Filesystem(harddrive, archinstall.GPT) as fs:
-    # We create a filesystem layout that will use the entire drive
-    # (this is a helper function, you can partition manually as well)
-    fs.use_entire_disk(root_filesystem_type='btrfs')
-
-    boot = fs.find_partition('/boot')
-    root = fs.find_partition('/')
-
-    boot.format('vfat')
-
-    # Set the flag for encrypted to allow for encryption and then encrypt
-    root.encrypted = True
-    root.encrypt(password=disk_password)
+## Scripting interactive installation
 
-with archinstall.luks2(root, 'luksloop', disk_password) as unlocked_root:
-    unlocked_root.format(root.filesystem)
-    unlocked_root.mount('/mnt')
+There are some examples in the `examples/` directory that should serve as a starting point.
 
-    boot.mount('/mnt/boot')
+The following is a small example of how to script your own *interative* installation:
 
-with archinstall.Installer('/mnt') as installation:
-    if installation.minimal_installation(hostname='minimal-arch'):
-        installation.add_bootloader()
-
-        installation.add_additional_packages(['nano', 'wget', 'git'])
-
-        # Optionally, install a profile of choice.
-        # In this case, we install a minimal profile that is empty
-        installation.install_profile('minimal')
-
-        user = User('devel', 'devel', False)
-        installation.create_users(user)
-        installation.user_set_pw('root', 'airoot')
+```python
+from pathlib import Path
+
+from archinstall import Installer, ProfileConfiguration, profile_handler, User
+from archinstall.default_profiles.minimal import MinimalProfile
+from archinstall.lib.disk.device_model import FilesystemType
+from archinstall.lib.disk.encryption_menu import DiskEncryptionMenu
+from archinstall.lib.disk.filesystem import FilesystemHandler
+from archinstall.lib.user_interaction.disk_conf import select_disk_config
+
+fs_type = FilesystemType('ext4')
+
+# Select a device to use for the installation
+disk_config = select_disk_config()
+
+# Optional: ask for disk encryption configuration
+data_store = {}
+disk_encryption = DiskEncryptionMenu(disk_config.device_modifications, data_store).run()
+
+# initiate file handler with the disk config and the optional disk encryption config
+fs_handler = FilesystemHandler(disk_config, disk_encryption)
+
+# perform all file operations
+# WARNING: this will potentially format the filesystem and delete all data
+fs_handler.perform_filesystem_operations()
+
+mountpoint = Path('/tmp')
+
+with Installer(
+        mountpoint,
+        disk_config,
+        disk_encryption=disk_encryption,
+        kernels=['linux']
+) as installation:
+    installation.mount_ordered_layout()
+    installation.minimal_installation(hostname='minimal-arch')
+    installation.add_additional_packages(['nano', 'wget', 'git'])
+
+    # Optionally, install a profile of choice.
+    # In this case, we install a minimal profile that is empty
+    profile_config = ProfileConfiguration(MinimalProfile())
+    profile_handler.install_profile_config(installation, profile_config)
+
+    user = User('archinstall', 'password', True)
+    installation.create_users(user)
 ```
 
 This installer will perform the following:
 
-* Prompt the user to select a disk and disk-password
-* Proceed to wipe the selected disk with a `GPT` partition table on a UEFI system and MBR on a BIOS system.
-* Sets up a default 100% used disk with encryption.
+* Prompt the user to configurate the disk partitioning
+* Prompt the user to setup disk encryption
+* Create a file handler instance for the configured disk and the optional disk encryption
+* Perform the disk operations (WARNING: this will potentially format the disks and erase all data)
 * Installs a basic instance of Arch Linux *(base base-devel linux linux-firmware btrfs-progs efibootmgr)*
 * Installs and configures a bootloader to partition 0 on uefi. On BIOS, it sets the root to partition 0.
 * Install additional packages *(nano, wget, git)*
+* Create a new user
 
 > **Creating your own ISO with this script on it:** Follow [ArchISO](https://wiki.archlinux.org/index.php/archiso)'s guide on how to create your own ISO.
 
+## Script non-interactive automated installation
+
+For an example of a fully scripted, automated installation please see the example
+[full_automated_installation.py](https://github.com/archlinux/archinstall/blob/master/examples/full_automated_installation.py)
+
 ## Unattended installation based on MAC address
 
-Archinstall comes with an [unattended](https://github.com/archlinux/archinstall/blob/master/examples/unattended.py) example which will look for a matching profile for the machine it is being run on, based on any local MAC address.
-For instance, if the machine that [unattended](https://github.com/archlinux/archinstall/blob/master/examples/unattended.py) is run on has the MAC address `52:54:00:12:34:56` it will look for a profile called [profiles/52-54-00-12-34-56.py](https://github.com/archlinux/archinstall/blob/master/profiles/52-54-00-12-34-56.py).
+Archinstall comes with an [unattended](https://github.com/archlinux/archinstall/blob/master/examples/mac_address_installation.py)
+example which will look for a matching profile for the machine it is being run on, based on any local MAC address.
+For instance, if the machine the code is executed on has the MAC address `52:54:00:12:34:56` it will look for a profile called
+[52-54-00-12-34-56.py](https://github.com/archlinux/archinstall/default_profiles/tailored.py).
 If it's found, the unattended installation will commence and source that profile as its installation procedure.
 
+# Profiles
+
+`archinstall` ships with a set of pre-defined profiles that can be chosen during the installation process.
+
+| *Desktop*     | *Server*   |
+|---------------|------------|
+| Awesome       | Cockpit    |
+| Bspwm         | Docker     |
+| Budgie        | Lighttpd   |
+| Cinnamon      | Mariadb    |
+| Cutefish      | Nginx      |
+| Deepin        | Postgresql |
+| Enlightenment | Tomcat     |
+| Gnome         | httpd      |
+| Kde           | sshd       |
+| Lxqt          |            |
+| Mate          |            |
+| Qtile         |            |
+| Sway          |            |
+| Xfce4         |            |
+| i3-wm         |            |
+
+The definitions of the profiles and what packages they will install can be seen directly in the menu or
+[default profiles](https://github.com/archlinux/archinstall/default_profiles)
+
+
 # Testing
 
 ## Using a Live ISO Image
@@ -167,8 +221,7 @@ you can replace the version of archinstall with a new version and run that with
 4. Now clone the latest repository with `git clone https://github.com/archlinux/archinstall`
 5. Enter the repository with `cd archinstall`
    *At this stage, you can choose to check out a feature branch for instance with `git checkout v2.3.1-rc1`*
-6. Build the project and install it using `python setup.py install`
-   *If you get a 'No Module named setuptools' error, run `pacman -S python-setuptools`*
+6. Build the project and install it using `pip install`
 
 After this, running archinstall with `python -m archinstall` will run against whatever branch you chose in step 5.