Please add support for local installation of SDK

[glaubitz]

On top to the versioning issues I ran into in #93, I also noticed that the project currently doesn't support running cargo install for local installation:

[ 1689s] + cd aws-sdk-rust-0.0.5+git+86cbc50
[ 1689s] + cd sdk
[ 1689s] + RUSTFLAGS='-Clink-arg=-Wl,-z,relro,-z,now -C debuginfo=2'
[ 1689s] + cargo install --root=/home/abuild/rpmbuild/BUILDROOT/aws-sdk-rust-0.0.5+git+86cbc50-8.1.x86_64/usr --path .
[ 1689s] warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
[ 1689s] package:   /home/abuild/rpmbuild/BUILD/aws-sdk-rust-0.0.5+git+86cbc50/sdk/examples/s3-helloworld/Cargo.toml
[ 1689s] workspace: /home/abuild/rpmbuild/BUILD/aws-sdk-rust-0.0.5+git+86cbc50/sdk/Cargo.toml
[ 1689s] error: found a virtual manifest at `/home/abuild/rpmbuild/BUILD/aws-sdk-rust-0.0.5+git+86cbc50/sdk/Cargo.toml` instead of a package manifest
[ 1689s] error: Bad exit status from /var/tmp/rpm-tmp.QvzBLQ (%install)

See: https://build.opensuse.org/package/live_build_log/home:glaubitz:branches:Cloud:Tools/aws-sdk-rust/openSUSE_Tumbleweed/x86_64

Would it be possible to add support for cargo install in the future so that the Rust SDK can be packaged for Linux distributions?

[rcoh]

typically, cargo install is only used for binary packages. The AWS Rust SDK is a library and would only be installable with cargo install when used as a dependency for another Rust project

[glaubitz]

Vendoring SDKs is rather suboptimal from a security point of view as it results in multiple copies of a library within the distribution.

See: https://fedoraproject.org/wiki/Bundled_Libraries?rd=Packaging:Bundled_Libraries

[mdaffin]

However rust does not yet support prebuilt dependencies. Currently this is just how rust works, every library is built into the binary. And looking at the ripgrep opensuse package it looks like it is vendoring all the deps inside the package rather than using external packages for its dependencies.

[glaubitz]

Well, it works in Debian which already ships tons of packages as libraries: https://qa.debian.org/developer.php?email=pkg-rust-maintainers%40alioth-lists.debian.net

My packaging was an attempt to expose the SDK faster to SUSE customers in the AWS Public Cloud.

[mdaffin]

Well, it works in Debian which already ships tons of packages as libraries: https://qa.debian.org/developer.php?email=pkg-rust-maintainers%40alioth-lists.debian.net

My packaging was an attempt to expose the SDK faster to SUSE customers in the AWS Public Cloud.

All those lib packages just look to contain the rust source files and only include the -dev version of the package. These are not binary shared libraries, only the source code so it can be used to build other rust binaries (which wont include these libs are separate runtime dependencies). This is likely done to capture the dependencies as they were at the time they were packaged. So all the source is contained in their packages rather than fetching them from upstream every time.

It seems that the opensuse ripgrep package is doing the same with a vendored download of its deps included inside the ripgrep source package (which is one hint to how opensuse might be trying to solve the problem).

If you really wanted to do this, it seems packaging the source would be the way to go. But I would look at other opensuse rust packages and see if they are all using this vendored approach - at which point your package would be the only one doing something different.

[glaubitz]

All those lib packages just look to contain the rust source files and only include the -dev version of the package. These are not binary shared libraries, only the source code so it can be used to build other rust binaries (which wont include these libs are separate runtime dependencies).

Yes, I'm aware of that. That doesn't invalidate my point. In Debian, the package are still compiled on each architecture and the testsuites are being run which guarantees the package works on every architecture that we support:

https://buildd.debian.org/status/package.php?p=rust-dbus&suite=sid

It seems that the opensuse ripgrep package is doing the same with a vendored download of its deps included inside the ripgrep source package (which is one hint to how opensuse might be trying to solve the problem).

openSUSE wants to get where Debian currently is for the very reasons that I mentioned earlier.

I just talked to my colleagues at SUSE earlier regarding this issue who have done the existing Rust packaging in openSUSE and they told me they are unhappy with the state of things where everything is being vendored.

If you really wanted to do this, it seems packaging the source would be the way to go. But I would look at other opensuse rust packages and see if they are all using this vendored approach - at which point your package would be the only one doing something different.

It won't be as the current state is not the state where openSUSE wants it to be. Distributions need to unvendor upstream packages as much as possible to be able to provide support to our customers and users.

[mdaffin]

All those lib packages just look to contain the rust source files and only include the -dev version of the package. These are not binary shared libraries, only the source code so it can be used to build other rust binaries (which wont include these libs are separate runtime dependencies).

Yes, I'm aware of that. That doesn't invalidate my point. In Debian, the package are still compiled on each architecture and the testsuites are being run which guarantees the package works on every architecture that we support:

https://buildd.debian.org/status/package.php?p=rust-dbus&suite=sid

It seems all that are doing for their test is essentially:

debian cargo wrapper: running subprocess (['env', 'RUST_BACKTRACE=1', '/usr/bin/cargo', '-Zavoid-dev-deps', 'build', '--verbose', '--verbose', '-j2', '--target', 's390x-unknown-linux-gnu'],) {}

https://buildd.debian.org/status/fetch.php?pkg=rust-dbus&arch=s390x&ver=0.9.0-3&stamp=1606959277&raw=0

And then throwing that away after the test and package only the source files into the -dev package. They are not doing a cargo install at all. Rust does not support that for libraries.

This is really an issue for packages to bring to the rust rather than to each library that might need to be packaged. Debian have their own policies for packages rust libraries and none of this requires rust libraries to do anything special to be packaged. I don't see why OpenSUSE would be any different in its requirements.

[glaubitz]

All those lib packages just look to contain the rust source files and only include the -dev version of the package. These are not binary shared libraries, only the source code so it can be used to build other rust binaries (which wont include these libs are separate runtime dependencies).

Yes, I'm aware of that. That doesn't invalidate my point. In Debian, the package are still compiled on each architecture and the testsuites are being run which guarantees the package works on every architecture that we support:

https://buildd.debian.org/status/package.php?p=rust-dbus&suite=sid

It seems all that are doing for their test is essentially:

debian cargo wrapper: running subprocess (['env', 'RUST_BACKTRACE=1', '/usr/bin/cargo', '-Zavoid-dev-deps', 'build', '--verbose', '--verbose', '-j2', '--target', 's390x-unknown-linux-gnu'],) {}

https://buildd.debian.org/status/fetch.php?pkg=rust-dbus&arch=s390x&ver=0.9.0-3&stamp=1606959277&raw=0

And then throwing that away after the test and package only the source files into the -dev package. They are not doing a cargo install at all. Rust does not support that for libraries.

Well, I can still hack something together on my own. But it's rather underwhelming when the overly praised Rust programming language doesn't support basic features that have been available in other programming languages for years.

This is really an issue for packages to bring to the rust rather than to each library that might need to be packaged. Debian have their own policies for packages rust libraries and none of this requires rust libraries to do anything special to be packaged. I don't see why OpenSUSE would be any different in its requirements.

Where did I say that openSUSE would be different from Debian in this regard. It was actually you who made that claim.

[MikailBag]

that have been available in other programming languages for years

Well, C++ has a similar problem AFAIK: if a library foo provides a generic class Bar, then you actually need Bar sources at the compile-time, and it is package libfoo has to contain Bar source code. For example:

# docker run -it --rm opensuse/leap
$ zypper install gcc gcc-c++
$ cat /usr/include/c++/vector
# source code of the STL vector implementation, because it can't be compiled and packaged as a .so

And in general, generic APIs are fundamentally incompatible with C-style linking, because generic functions and types must be instantiated into concrete items. And compiler can't perform this instantiation when it compiles the library, because at this time it doesn't know which instantiation will be requested by the application. And you can see that SDK crates actually use generics (e.g. client is generic over C: SmithyConnector).

That's why I think if you really want to package AWS SDK as a set of .so-files, some non-trivial work (at minimum either patching SDK or providing generic-free wrappers) is required.

[glaubitz]

that have been available in other programming languages for years

Well, C++ has a similar problem AFAIK: if a library foo provides a generic class Bar, then you actually need Bar sources at the compile-time, and it is package libfoo has to contain Bar source code. For example:

# docker run -it --rm opensuse/leap
$ zypper install gcc gcc-c++
$ cat /usr/include/c++/vector
# source code of the STL vector implementation, because it can't be compiled and packaged as a .so

That's a false equivalency. There are enough C++ projects that are shipped as shared libraries such as Qt.

And in general, generic APIs are fundamentally incompatible with C-style linking, because generic functions and types must be instantiated into concrete items.

I don't think this applies to the whole API that this SDK provides.

That's why I think if you really want to package AWS SDK as a set of .so-files, some non-trivial work (at minimum either patching SDK or providing generic-free wrappers) is required.

I was able to package the C++ version of the SDK without any problems. If Rust is supposed to be able to compete with C/C++, this should work there as well.

Distributions need to use shared libraries to avoid code copies on disk. Otherwise, security support becomes unfeasible.

[MikailBag]

There are enough C++ projects that are shipped as shared libraries such as Qt

AFAIK, Qt developers made a substantial amount of work to make it possible. And still the following code: https://github.com/qt/qtbase/blob/dev/src/corelib/tools/qlist.h#L99 has to be available in the include paths (because it is used by the #include <QVector>).

I don't think this applies to the whole API that this SDK provides.

As I already said, some APIs are generic. If you want to compile this SDK into a .so, all such APIs have to be wrapped by the non-generic ones (at least I don't see any other way to go). If at least one API is generic, its source code has to packaged and statically link into all binary packages. In the case of the Rust SDK, it is the Client which is the heart of the library.

I was able to package the C++ version of the SDK without any problems.

As I see, C++ SDK does not use templates, but Rust one uses.

Distributions need to use shared libraries to avoid code copies on disk.

I understand it is easier for a distribution maintainer to rebuild one library package rather than all binary packages using those libraries. I just want to say it is pretty hard to achieve.

To be honest, I'm pretty sure this was already discussed somewhere in the rust-lang/rust (with far more details and arguments I can ever provide), but I can't find any relevant issue, sorry.

[glaubitz]

AFAIK, Qt developers made a substantial amount of work to make it possible. And still the following code: https://github.com/qt/qtbase/blob/dev/src/corelib/tools/qlist.h#L99 has to be available in the include paths (because it is used by the #include ).

I understand it is easier for a distribution maintainer to rebuild one library package rather than all binary packages using those libraries. I just want to say it is pretty hard to achieve.

It would already be enough when I could build and install the package with cargo without hacking something together myself. It doesn't necessarily have to be binary-installable as long as I can have one canonical SDK installation which can be sourced by all consuming projects.

In openSUSE/SLE, the build system will automatically rebuild all reverse dependencies of a package if the package gets updated. So if I can create an RPM package from the Rust SDK, it can be used by other Rust projects in openSUSE/SLE and they will still only build against one particular source version of the package and automatically get rebuild if I update the SDK.

Without that, I would have to manually update every single package that uses the SDK which isn't really a preferred solution.

[mdaffin]

It doesn't necessarily have to be binary-installable as long as I can have one canonical SDK installation which can be sourced by all consuming projects.

In openSUSE/SLE, the build system will automatically rebuild all reverse dependencies of a package if the package gets updated. So if I can create an RPM package from the Rust SDK, it can be used by other Rust projects in openSUSE/SLE and they will still only build against one particular source version of the package and automatically get rebuild if I update the SDK.

Without that, I would have to manually update every single package that uses the SDK which isn't really a preferred solution.

You can do this already. Just put the source files in the package at the right location like the debian packages do. There is no need for this or any rust library to do anything special to make it packagable as a set of source files.

[github-actions]

Greetings! Sorry to say but this is a very old issue that is probably not getting as much attention as it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to open a new one.