Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Binary releases #1

Closed
drewcrawford opened this issue Feb 6, 2016 · 9 comments

Comments

@drewcrawford
Copy link
Contributor

commented Feb 6, 2016

Some of my dependency chains are long enough now that compile times are a real issue.

We should support the downloading and linking of precompiled binaries, if available.

This may require atbuild/atpkg support.

@owensd

This comment has been minimized.

Copy link
Member

commented Feb 6, 2016

Yes, yes we should. :)

Sent from my iPhone

On Feb 5, 2016, at 6:58 PM, Drew Crawford notifications@github.com wrote:

Some of my dependency chains are long enough now that compile times are a real issue.

We should support the downloading and linking of precompiled binaries, if available.

This may require atbuild/atpkg support.


Reply to this email directly or view it on GitHub.

@drewcrawford

This comment has been minimized.

Copy link
Contributor Author

commented Apr 26, 2016

Been doing a little brainstorming about this, as I have a project where source isn't distributed.

I think we need a manifest format, something like

(package 
:name "foo"
:releases {
    :1.0 {
        :url "https://myhost/foo-1.0.tar.gz"
    }
    :0.10.0 {
        :url "https://myhost/foo-0.10.0.tar.gz"
    }
})

Then in your external-packages you can specify

  :external-packages [
    {
      :url "https://sealedabstract.com/foo.atpkg"
      :version [ "1.0.0" ]
    }
  ]

which would fetch the manifest over HTTPS (e.g. not with git), then find the appropriate tar, and then fetch/uncompress that.

Additional thoughts:

  1. Plaintext HTTP should probably be disabled by default, although allowed with an option (--insecure?)
  2. Binary releases should also be supported when we learn about them from a source repository instead of an HTTPS manifest. I guess there should be some option whether to prefer building from source or binary release when both are available.
  3. atbuild will need to be extended to support libraries not built from source, which it currently doesn't
  4. Binaries may be only available on private servers (a common case for me). I'm not totally sure what the right conclusion here is, options include distribution via private S3 bucket or Git LFS.

I can potentially contrib this eventually if nobody else is motivated, as the binary-only thing is an actual case for me

@dunkelstern

This comment has been minimized.

Copy link
Member

commented Apr 27, 2016

For that to work i need a working HTTPs client or shelling out to curl (which doesn't seem ideal) but yeah, seems doable.

I think it should look like this:

  :external-packages [
    {
      :url "https://sealedabstract.com/foo.git"
      :version [ "1.0.0" ]
      :prefer-precompiled true
    }
  ]

I would prefer to always have a git repository at the url. It could have either code + manifest file or only one of those. So you can speed up building by enabling the flag but switch back to building from source momentarily (and you have a place to look for a release log).

Because we don't have an ABI contract for swift yet the "repository" for the precompiled stuff will get a rather complicated thing. Perhaps the best thing would be to have tooling that creates that repo for you so no human error is involved.

If that is the correct way to do it we should keep the manifest separate from the build.atpkg as the tooling would rewrite the manifest file. Could look like this:

(manifest
  :name "foo"
  :url "https://sealedabstract.com/foo.git"
  :releases {
    :1.0 {
      :url "https://sealedabstract.com/foo-1.0.bin.${swift_version}.tar.gz"
      :supported-swift-versions [ "3.0-dev-36739f7b57" ]
    }
    :1.2 {
      :url "https://sealedabstract.com/foo-1.2.bin.${swift_version}.tar.gz"
      :supported-swift-versions [ "3.0-dev-36739f7b57" ]
    }
  }
)

Alternatively we could let it build the path for you:

(manifest
  :name "foo"
  :url "https://sealedabstract.com/foo.git"
  :releases {
    :base-url "https://sealedabstract.com/foo-${version}.bin.${swift_version}.tar.gz"
    :supported-swift-versions [ "3.0-dev-36739f7b57" ]

    :1.2 {
      :url "https://specialsnowflake.com/foo-1.2.bin.${swift_version}.tar.gz"
    }

    :0.9 {
      :supported-swift-versions [ "2.0" "2.1" "2.1.1" "2.2" ]
    }
  }
)

Where the supported-swift-versions is inherited by the 1.2 override and the base-url is inherited by the 0.9 override.

I propose the command atpm precompile [--toolchain=XXX] for building the archive files. Probably with multiple --toolchain= arguments to build multiple versions at once.

@dunkelstern

This comment has been minimized.

Copy link
Member

commented Apr 27, 2016

Those manifest files could even be collected into some sort of package index for people to search (like the IBM thing for SwiftPM)

@drewcrawford

This comment has been minimized.

Copy link
Contributor Author

commented Apr 27, 2016

I would prefer to always have a git repository at the url. It could have either code + manifest file or only one of those.

There is no published git repository for any of my binary-only libraries. IMO the requirement to host one is prohibitive to using the feature. This is a mistake that e.g. Carthage is making.

HTTP is just a lot simpler to host than git. For sourcecode git makes sense, for one manifest file it usually doesn't. Of course, nobody should stop you from being able to use git if you want to for hosting your manifest, I just don't think it should be a requirement, as it's more convoluted than copying a file up to S3.

For that to work i need a working HTTPs client or shelling out to curl (which doesn't seem ideal) but yeah, seems doable.

We need to solve this problem internally, so if it's just a question of implementation complexity, we can contrib our internal solution after we build it. It's not clear at this point what it will be, but we will build something and can make it available if that unlocks a problem for AT.

Because we don't have an ABI contract for swift yet the "repository" for the precompiled stuff will get a rather complicated thing.

This is a good insight, I had not considered this. But it's not clear to me what the actionable part of this is, e.g. how do we know which swift will be used to build? I suppose we could pass on the CLI (like --toolchain in atbuild).

The other actionable part is what we actually do with this data. e.g., will there really be multiple binary ABI versions to choose from? I'm a pretty large binary user and I only build binaries against one Swift (although it may vary per-platform, e.g. linux on Swift3 with Darwin/iOS on Swift 2 is a common config for me). But I usually don't support Swift3/2 on the same platform. So I don't know if this unlocks enough of a problem to justify the complexity.

[ "3.0-dev-36739f7b57" ] is probably not a good descriptor for Swift 3.0, as CaffeinatedSwift is compatible with the corresponding Swift snapshot even though it has a different commit for the out-of-tree patches. So we probably need something like tags here for this to be useful.

@dunkelstern

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

It's rather hard to determine a git repo by just looking at an url... Either we would have to go the python pip way of using git+https or we have to rename the key (which i would prefer).

@drewcrawford

This comment has been minimized.

Copy link
Contributor Author

commented Apr 28, 2016

Do git URLs not end in git?

If not, a key would not be a great hardship. Just news to me 'sall

@dunkelstern

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

That .git that github uses is purely cosmetic, if you omit it it will work too. I work on a proposal and post it here when it's ready.

@drewcrawford

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2016

@dunkelstern I have time to work on this, this week.

drewcrawford added a commit to AnarchyTools/atpkg that referenced this issue May 28, 2016
WIP: atpkg support for binary atpm
See AnarchyTools/atpm#1

This commit adds

* Dependency type to ExternalDependency, to distinguish between Git and Manifest types
  This design was settled on after looking at several, including autodetecting git (which is possible via git ls-remote)
  However, manifest ExternalDependency is quite different than Git ExternalDependency for several reasons (including the use of channels)
* Channels, which are essentially like micropackages inside a package.  A channel lets us say "stable" "beta" "linux" "osx" "swift2.2" etc.  All releases are part of a channel.  It would be possible to use separate manifests for each channel, but this would require the use of multiple manifests for basic osx/linux packages, which is undesireable
* Tokens can now start with numbers, which is important, as we now have syntax where `:1.0 { :url "https://example.com/some/tarball.xz"}` is used to define where some version can be located
drewcrawford added a commit to AnarchyTools/atbuild that referenced this issue Jun 7, 2016
Add the ability to link with an atbin
Resolve #13

This PR introduces `link-with-atbin` which allows linking with an arbitrary atbin (such as a binary release, see AnarchyTools/atpm#1) or any other atbin you have lying around.

`link-with` has been renamed `link-with-product` and we intend to deprecate the old syntax.
drewcrawford added a commit to AnarchyTools/atpkg that referenced this issue Jun 30, 2016
WIP: atpkg support for binary atpm
See AnarchyTools/atpm#1

This commit adds

* Dependency type to ExternalDependency, to distinguish between Git and Manifest types
  This design was settled on after looking at several, including autodetecting git (which is possible via git ls-remote)
  However, manifest ExternalDependency is quite different than Git ExternalDependency for several reasons (including the use of channels)
* Channels, which are essentially like micropackages inside a package.  A channel lets us say "stable" "beta" "linux" "osx" "swift2.2" etc.  All releases are part of a channel.  It would be possible to use separate manifests for each channel, but this would require the use of multiple manifests for basic osx/linux packages, which is undesireable
* Tokens can now start with numbers, which is important, as we now have syntax where `:1.0 { :url "https://example.com/some/tarball.xz"}` is used to define where some version can be located
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.