Snapcraft v7 fails to build core18 snap with local plugins

[pachulo]

Bug Description

Trying to build the nextcloud-snap, which uses local plugins, with snapcraft v7.x fails.
It works OK with snapcraft v6.x

See https://forum.snapcraft.io/t/snapcraft-7-refuses-to-build-core18-snap/30592 for more context.

To Reproduce

Try to build the nextcloud-snap with snapcraft >= 7.0

Environment

Using LXD on Ubuntu 20.04 LTS.

snapcraft.yaml

name: nextcloud
version: git
summary: Nextcloud Server - A safe home for all your data
description: |
 Access, share and protect your files, calendars, contacts, communication and
 more at home and in your enterprise.

grade: stable
confinement: strict
base: core18

architectures:
  - build-on: amd64
  - build-on: i386
  - build-on: arm64
  - build-on: armhf
  - build-on: ppc64el

apps:
  # Apache daemon
  apache:
    command: bin/run-httpd -k start -DFOREGROUND
    stop-command: bin/httpd-wrapper -k stop
    reload-command: bin/httpd-wrapper -k graceful
    daemon: simple
    restart-condition: always
    plugs: [network, network-bind, removable-media]

  # MySQL daemon
  mysql:
    command: bin/start_mysql
    stop-command: support-files/mysql.server stop
    reload-command: bin/reload-mysql
    daemon: simple
    restart-condition: always
    plugs: [network, network-bind]

  # PHP FPM daemon
  php-fpm:
    command: bin/start-php-fpm
    reload-command: bin/reload-php
    daemon: simple
    restart-condition: always
    plugs:
      - network
      - network-bind

      # Optional interface for observing network in system settings
      - network-observe

      # Optional interface for accessing removable media
      - removable-media

  # redis server daemon
  redis-server:
    command: bin/start-redis-server
    daemon: simple
    restart-condition: always
    plugs: [network, network-bind]

  # MySQL client
  mysql-client:
    command: bin/run-mysql
    plugs: [network, network-bind]

  mysqldump:
    command: bin/run-mysqldump
    plugs: [network, network-bind]

  # Nextcloud occ command
  occ:
    command: bin/occ
    plugs: [network, network-bind, removable-media]

  enable-https:
    command: bin/enable-https
    plugs: [network, network-bind]

  disable-https:
    command: bin/disable-https
    plugs: [network, network-bind]

  renew-certs:
    command: bin/renew-certs
    daemon: simple
    restart-condition: always
    plugs: [network, network-bind]

  nextcloud-cron:
    command: bin/nextcloud-cron
    daemon: simple
    restart-condition: on-failure
    plugs: [network, network-bind, removable-media]

  # Command for manually installing instead of visiting site to create admin.
  manual-install:
    command: bin/manual-install
    plugs: [network, network-bind, removable-media]

  import:
    command: bin/import-data
    plugs: [network, network-bind, removable-media]

  export:
    command: bin/export-data
    plugs: [network, network-bind, removable-media]

  # Service for automatically fixing Nextcloud indices, etc.
  nextcloud-fixer:
    command: bin/nextcloud-fixer
    daemon: simple
    restart-condition: on-failure
    plugs: [network, network-bind, removable-media]

  logrotate:
    command: bin/run-logrotate
    daemon: simple
    restart-condition: on-failure
    timer: 00:00 # Run once a day at midnight

hooks:
  configure:
    plugs: [network, network-bind, removable-media]
  pre-refresh:
    plugs: [network, network-bind, removable-media]

parts:
  apache:
    plugin: apache
    source: https://dlcdn.apache.org/httpd/httpd-2.4.57.tar.bz2
    source-checksum: sha256/dbccb84aee95e095edfbb81e5eb926ccd24e6ada55dcd83caecb262e5cf94d2a
    
    build-packages:
      - libbrotli-dev

    stage-packages:
      - libbrotli1

      # The built-in Apache modules to enable
    modules:
      - headers
      - proxy
      - proxy_fcgi
      - setenvif
      - env
      - rewrite
      - mime
      - dir
      - authz_core
      - unixd
      - alias
      - ssl
      - socache_shmcb
      - slotmem_shm
      - log_config
      - logio
      - brotli
      - filter

    filesets:
      exclude:
        - -man
        - -manual
        - -htdocs
        - -include
        - -build
        - -conf/httpd.conf
        - -conf/magic
        - -conf/original
        - -conf/extra
        - -bin/apachectl
        - -bin/envvars*
    stage:
      - $exclude
    prime:
      - $exclude

  apache-customizations:
    plugin: dump
    source: src/apache/

  nextcloud:
    plugin: dump
    source: https://download.nextcloud.com/server/releases/nextcloud-26.0.3.tar.bz2
    source-checksum: sha256/a5a810cbcd7c3dcff25f22809321f2ed41ed7cc0601204489ac90e258060b5c9
    organize:
      '*': htdocs/
      '.htaccess': htdocs/.htaccess
      '.user.ini': htdocs/.user.ini

    # This snap automatically updates. No need to include the updater to nag
    # users. This does not result in an integrity check failure.
    prime:
      - -htdocs/apps/updatenotification

  php:
    plugin: php
    source: https://php.net/get/php-8.1.20.tar.bz2/from/this/mirror
    source-checksum: sha256/55578587514a2707500f85319e57c0d4df9b8803cdb26566595ac4bf459dc4dd
    source-type: tar
    install-via: prefix
    configflags:
      - --enable-fpm
      - --disable-cgi
      - --disable-phar
      - --disable-phpdbg
      - --enable-ctype
      - --enable-mbstring
      - --with-zip
      - --with-pdo-mysql
      - --with-zlib
      - --enable-gd
      - --with-curl
      - --with-openssl
      - --with-bz2
      - --enable-exif
      - --enable-intl
      - --enable-pcntl
      - --with-jpeg
      - --with-freetype
      - --disable-rpath
      - --enable-ftp
      - --enable-bcmath
      - --enable-sysvsem

      # Enable ldap
      - --with-libdir=lib/$SNAPCRAFT_ARCH_TRIPLET
      - --with-ldap

      # Enable gmp
      - --with-gmp

      # Enable argon2
      - --with-password-argon2

      # Disable sqlite (we use mysql)
      - --without-sqlite3
      - --without-pdo-sqlite
    build-packages:
      - libxml2-dev
      - libcurl4-openssl-dev
      - libpng-dev
      - libjpeg8-dev
      - libbz2-dev
      - libmcrypt-dev
      - libldap2-dev
      - libfreetype6-dev
      - libgmp-dev
      - libzip-dev
      - libargon2-0-dev

      # This is no longer bundled with PHP as of v7.4
      - libonig-dev
    stage-packages:
      - libasn1-8-heimdal
      - libcurl4
      - libfreetype6
      - libgssapi3-heimdal
      - libhcrypto4-heimdal
      - libheimbase1-heimdal
      - libheimntlm0-heimdal
      - libhx509-5-heimdal
      - libicu60
      - libjpeg8
      - libkrb5-26-heimdal
      - libldap-2.4-2
      - libnghttp2-14
      - libpng16-16
      - libpsl5
      - libroken18-heimdal
      - librtmp1
      - libsasl2-2
      - libwind0-heimdal
      - libxml2
      - libzip4
      - libargon2-0
      - libonig4
    prime:
     - -sbin/
     - -etc/
    organize:
      # Put php-fpm in bin with everything else, not sbin.
      sbin/php-fpm: bin/php-fpm
    extensions:
      # Build the redis PHP module
      - source: https://github.com/phpredis/phpredis/archive/5.3.7.tar.gz
        source-checksum: sha256/6f5cda93aac8c1c4bafa45255460292571fb2f029b0ac4a5a4dc66987a9529e6

  redis:
    plugin: redis
    source: https://download.redis.io/releases/redis-6.2.12.tar.gz
    source-checksum: sha256/75352eef41e97e84bfa94292cbac79e5add5345fc79787df5cbdff703353fb1b

  redis-customizations:
    plugin: dump
    source: src/redis/
    organize:
      config/*: config/redis/
    after: [envsubst]

  # Copy over our PHP configuration file.
  php-customizations:
    plugin: dump
    source: src/php/
    organize:
      config/*: config/php/
    stage-packages: [mawk]

  # Copy over our Nextcloud configuration files
  nextcloud-customizations:
    plugin: dump
    source: src/nextcloud/
    organize:
      config/: htdocs/config/
    stage-packages:
      # jq is needed for parsing occ's json output
      - jq

      # The collabora appimage requires this
      - libfontconfig1

  boost:
    plugin: dump
    source: https://sourceforge.net/projects/boost/files/boost/1.77.0/boost_1_77_0.tar.bz2
    source-checksum: sha1/0cb4f947d094fc311e13ffacaff00418130ef5ef
    stage:
      - boost/
    prime:
      - -*

  mysql:
    plugin: cmake
    after: [boost]

    # Get from https://dev.mysql.com/downloads/mysql/
    source: https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-boost-8.0.33.tar.gz
    source-checksum: md5/20ffc71fb8acd705cdc4a8ae4cdedf23
    configflags:
      - -DCMAKE_INSTALL_PREFIX=/
      - -DBUILD_CONFIG=mysql_release
      - -DCMAKE_BUILD_TYPE=Release
      - -DWITH_UNIT_TESTS=OFF
      - -DWITH_EMBEDDED_SERVER=OFF
      - -DWITH_ARCHIVE_STORAGE_ENGINE=OFF
      - -DWITH_EXAMPLE_STORAGE_ENGINE=OFF
      - -DWITH_BLACKHOLE_STORAGE_ENGINE=OFF
      - -DWITH_FEDERATED_STORAGE_ENGINE=OFF
      - -DWITH_PARTITION_STORAGE_ENGINE=OFF
      - -DWITH_PERFSCHEMA_STORAGE_ENGINE=OFF
      - -DWITH_DEBUG=OFF
      - -DWITH_INNODB_EXTRA_DEBUG=OFF
      - -DINSTALL_MYSQLTESTDIR=
      - -DDOWNLOAD_BOOST=OFF
      - -DWITH_BOOST=$SNAPCRAFT_STAGE

    override-build: |
      snapcraftctl build
      # MySQL v8 is massive. Strip it.
      find "$SNAPCRAFT_PART_INSTALL/bin" -type f -exec sh -c 'grep -IL . "$1" || strip --strip-all "$1"' sh "{}" \;

    build-packages:
      - libncurses5-dev
      - binutils
      - gcc-8
      - g++-8

    stage:
      # Remove scripts that we'll be replacing with our own
      - -support-files/mysql.server
      - -COPYING

      # Remove unused binaries that waste space
      - -bin/comp_err
      - -bin/ibd2sdi
      - -bin/innochecksum
      - -bin/lz4_decompress
      - -bin/myisam_ftdump
      - -bin/myisam*
      - -bin/mysql_client_test
      - -bin/mysql_config*
      - -bin/mysql_secure_installation
      - -bin/mysql_ssl_rsa_setup
      - -bin/mysql_tzinfo_to_sql
      - -bin/mysql_upgrade
      - -bin/mysqladmin
      - -bin/mysqlbinlog
      - -bin/mysqlcheck
      - -bin/mysqld_multi
      - -bin/mysqlimport
      - -bin/mysqlpump
      - -bin/mysqlrouter*
      - -bin/mysqlshow
      - -bin/mysqlslap
      - -bin/mysqltest*
      - -bin/perror
      - -bin/zlib_decompress

  # Copy over our MySQL scripts
  mysql-customizations:
    plugin: dump
    source: src/mysql/

  patches:
    source: src/patches
    plugin: dump
    prime:
      - -*

  delay-on-failure:
    plugin: dump
    source: src/delay-on-failure/

  certbot:
    plugin: python
    python-version: python2
    source: src/https/
    requirements: ["requirements.txt"]
    build-packages: [libffi-dev]
    after: [patches]
    override-build: |
      snapcraftctl build
      patch -p1 -d $SNAPCRAFT_PART_INSTALL/lib/python2.7/site-packages/certbot < $SNAPCRAFT_STAGE/certbot-remove-default-config-files.patch
      patch -p1 -d $SNAPCRAFT_PART_INSTALL/lib/python2.7/site-packages/certbot < $SNAPCRAFT_STAGE/certbot-remove-storage-chown.patch

  setup-https:
    plugin: dump
    source: src/https/
    stage-packages: [openssl]
    stage: [-etc/ssl, -requirements.txt]

  import-export:
    plugin: dump
    source: src/import-export
    stage-packages: [rsync]

  common:
    plugin: dump
    source: src/common/

  hooks:
    plugin: dump
    source: src/hooks/
    organize:
      bin/: snap/hooks/
    stage-packages: [curl]

  logrotate:
    plugin: dump
    source: src/logrotate/
    organize:
      config/*: config/logrotate/
      usr/sbin/*: bin/
    stage-packages: [logrotate]
    stage:
      - bin/*
      - config/*
      - utilities/*
    after: [envsubst]

  migrations:
    plugin: dump
    source: src/migrations/

  envsubst:
    plugin: nil
    stage-packages: [gettext-base]
    stage: [usr/bin/envsubst]

Relevant log output

Sorry, an error occurred in Snapcraft:
module 'snapcraft' has no attribute 'sources'
Traceback (most recent call last):
  File "/snap/snapcraft/7720/lib/python3.8/site-packages/snapcraft/cli.py", line 188, in run
    dispatcher.run()
  File "/snap/snapcraft/7720/lib/python3.8/site-packages/craft_cli/dispatcher.py", line 406, in run
    return self._loaded_command.run(self._parsed_command_args)
  File "/snap/snapcraft/7720/lib/python3.8/site-packages/snapcraft/commands/lifecycle.py", line 82, in run
    parts_lifecycle.run(self.name, parsed_args)
  File "/snap/snapcraft/7720/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 135, in run
    yaml_data = process_yaml(snap_project.project_file)
  File "/snap/snapcraft/7720/lib/python3.8/site-packages/snapcraft/parts/lifecycle.py", line 86, in process_yaml
    yaml_data = yaml_utils.load(yaml_file)
  File "/snap/snapcraft/7720/lib/python3.8/site-packages/snapcraft/parts/yaml_utils.py", line 92, in load
    raise errors.LegacyFallback("base is not core22")
snapcraft.errors.LegacyFallback: base is not core22

Additional context

No response

[tigarmo]

Looks like we need to redirect the import done here

[cmatsuoka]

Correct, Snapcraft 7 redirects selected legacy imports to snapcraft_legacy. This is currently done with snapcraft.plugins.v1 and snapcraft.plugins.v2, however the php local plugin in this project invokes snapcraft.sources.get which is not redirect. Handling this should fix this issue, however local plugins importing other symbols from snapcraft may fail.

[pachulo]

Just tested the building of nextcloud-snap with snapcraft 7.5.3 and it succeeded! Thanks for this!