New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new-sdist command #5389

Merged
merged 19 commits into from Jun 25, 2018

Conversation

Projects
4 participants
@typedrat
Collaborator

typedrat commented Jun 19, 2018

new-sdist is done, and so are the tests (thanks @quasicomputational). There are some unfortunate implications I don't see a way around in terms of how new-style builds work, meaning that the sdist hooks are dead. Fortunately, I don't see many cases this will actually effect. I only found three packages in a representative sample that would be "broken" (at sdist time, has no effects on end users of these packages) and honestly it'd not be much sweat off my back to fix them myself.

Status:

  • .tar.gz sdists
  • .zip sdists
  • Listing source files
  • Works with custom setup (in a sense)

Closes #4293, but does not touch #4047 or #4675, previously thought to be required. It solves the issue by simply not running preprocessors and not running Setup configure under the logic that whoever is downloading this sdist will be able to do it themselves in all likelihood. Doing otherwise would make it impossible to insure that the same source files will generate an identical sdist, which is required for new-install to work.

Please include the following checklist in your PR:

  • Patches conform to the coding conventions.
  • Any changes that could be relevant to users have been recorded in the changelog.
  • The documentation has been updated, if necessary.
  • If the change is docs-only, [ci skip] is used to avoid triggering the build bots.

@typedrat typedrat self-assigned this Jun 19, 2018

@typedrat typedrat requested review from 23Skidoo and hvr Jun 19, 2018

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 19, 2018

Collaborator

It is stable at least on the same machine:

❯ shasum -a 256 dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz ../test/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
1e70454db3e883ef791fc851c63933d0017c790b0f592d7b9e7efeaf8a144042  dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
1e70454db3e883ef791fc851c63933d0017c790b0f592d7b9e7efeaf8a144042  ../test/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
Collaborator

typedrat commented Jun 19, 2018

It is stable at least on the same machine:

❯ shasum -a 256 dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz ../test/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
1e70454db3e883ef791fc851c63933d0017c790b0f592d7b9e7efeaf8a144042  dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
1e70454db3e883ef791fc851c63933d0017c790b0f592d7b9e7efeaf8a144042  ../test/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
@hvr

This comment has been minimized.

Show comment
Hide comment
@hvr

hvr Jun 19, 2018

Member

@23Skidoo this is kinda orthogonal, but do you know who actually benefits from .zip support? and why we're keeping it around? Afaik we don't support package repos with .zip files and I wonder if cabal install pkg-1.2.3.zip ever worked...

Member

hvr commented Jun 19, 2018

@23Skidoo this is kinda orthogonal, but do you know who actually benefits from .zip support? and why we're keeping it around? Afaik we don't support package repos with .zip files and I wonder if cabal install pkg-1.2.3.zip ever worked...

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 19, 2018

Collaborator

Either way, it's there now at the cost of four (soon to be two, once jgm/zip-archive#47 goes through) dependencies for cabal-install.

Collaborator

typedrat commented Jun 19, 2018

Either way, it's there now at the cost of four (soon to be two, once jgm/zip-archive#47 goes through) dependencies for cabal-install.

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 19, 2018

Collaborator

Formally note down:

  • Invariant that custom Setups can't change files in such a way that the file generated by cabal new-sdist changes
  • Distribution.Simple.UserHooks.UserHooks{preSDist, sDistHook, postSDist} are/will be deprecated now, because they interfere with the Glorious Deterministic Future. A replacement is needed, probably some sort of script hooks.
Collaborator

typedrat commented Jun 19, 2018

Formally note down:

  • Invariant that custom Setups can't change files in such a way that the file generated by cabal new-sdist changes
  • Distribution.Simple.UserHooks.UserHooks{preSDist, sDistHook, postSDist} are/will be deprecated now, because they interfere with the Glorious Deterministic Future. A replacement is needed, probably some sort of script hooks.
@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 19, 2018

Collaborator

I need to add tests for new-sdist, to confirm the determinicity.

Collaborator

typedrat commented Jun 19, 2018

I need to add tests for new-sdist, to confirm the determinicity.

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 19, 2018

Collaborator

To a first approximation, if the sdist hooks completely disppeared tomorrow (and obviously it's more gradual than that), the following (actively maintained and used enough to get into certain curated systems) packages would break:

  • gl-0.8.0
  • hlibsass-0.1.6.1
  • ShellCheck-0.4.7

I've basically managed to fix gl, I'd try the other two but I do need some sleep. That's 0.12% of the packages in my sample (latest Stackage LTS).

Collaborator

typedrat commented Jun 19, 2018

To a first approximation, if the sdist hooks completely disppeared tomorrow (and obviously it's more gradual than that), the following (actively maintained and used enough to get into certain curated systems) packages would break:

  • gl-0.8.0
  • hlibsass-0.1.6.1
  • ShellCheck-0.4.7

I've basically managed to fix gl, I'd try the other two but I do need some sleep. That's 0.12% of the packages in my sample (latest Stackage LTS).

typedrat added some commits Jun 19, 2018

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 20, 2018

Collaborator

While I understand there are some weighty elements to this, at the same time it is blocking new-install.

Collaborator

typedrat commented Jun 20, 2018

While I understand there are some weighty elements to this, at the same time it is blocking new-install.

@typedrat typedrat changed the title from [WIP] Add new-sdist command. to Add new-sdist command Jun 20, 2018

@quasicomputational

Looks good. Left some comments but they're just little niggles. Wouldn't mind having a test exercising --list-only, and maybe also the non-straightforward target selector cases: multiple valid selectors, all, some valid + an invalid one, and exercise all the invalid cases.

No comment on the over-all design or the hooks stuff, which I don't understand well enough to have an informed opinion on.

Show outdated Hide outdated cabal-install/Distribution/Client/CmdSdist.hs
Show outdated Hide outdated cabal-install/Distribution/Client/CmdSdist.hs
Show outdated Hide outdated cabal-install/Distribution/Client/CmdSdist.hs
[ optionVerbosity
sdistVerbosity (\v flags -> flags { sdistVerbosity = v })
, optionDistPref
sdistDistDir (\dd flags -> flags { sdistDistDir = dd })

This comment has been minimized.

@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

This is claiming its default is dist, but it actually plops the tarball in dist-newstyle.

@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

This is claiming its default is dist, but it actually plops the tarball in dist-newstyle.

This comment has been minimized.

@typedrat

typedrat Jun 20, 2018

Collaborator

That's #5380 and it's its own issue. optionDistPref needs fixing, but that's neither here nor there for this patchset.

@typedrat

typedrat Jun 20, 2018

Collaborator

That's #5380 and it's its own issue. optionDistPref needs fixing, but that's neither here nor there for this patchset.

Show outdated Hide outdated cabal-install/Distribution/Client/CmdSdist.hs
@@ -261,6 +261,15 @@ HACKAGE_SECURITY_VER="0.5.3.0"; HACKAGE_SECURITY_VER_REGEXP="0\.5\.((2\.[2-9]|[3
# >= 0.5.2 && < 0.6
TAR_VER="0.5.1.0"; TAR_VER_REGEXP="0\.5\.([1-9]|1[0-9]|0\.[3-9]|0\.1[0-9])\.?"
# >= 0.5.0.3 && < 0.6
# These two are temporary: https://github.com/jgm/zip-archive/pull/47

This comment has been minimized.

@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

Not actually a blocker for merging, right? I'd be OK with merging as-is and then fixing up when the PR is merged upstream.

@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

Not actually a blocker for merging, right? I'd be OK with merging as-is and then fixing up when the PR is merged upstream.

This comment has been minimized.

@typedrat

typedrat Jun 20, 2018

Collaborator

Nope, not a blocker. That's to remind us when it does get merged (hopefully, I don't see a reason it wouldn't be).

@typedrat

typedrat Jun 20, 2018

Collaborator

Nope, not a blocker. That's to remind us when it does get merged (hopefully, I don't see a reason it wouldn't be).

@@ -282,6 +282,7 @@ cabalG' global_args cmd args = do
| cmd `elem` ["v1-update", "outdated", "user-config", "manpage", "v1-freeze", "check"]
= [ ]
-- new-build commands are affected by testCabalProjectFile
| cmd == "new-sdist" = [ "--project-file", testCabalProjectFile env ]

This comment has been minimized.

@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

Isn't this covered by the next case, catching all the new- commands?

@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

Isn't this covered by the next case, catching all the new- commands?

This comment has been minimized.

@typedrat

typedrat Jun 20, 2018

Collaborator

It can’t take and is confused by ‘-j1’.

@typedrat

typedrat Jun 20, 2018

Collaborator

It can’t take and is confused by ‘-j1’.

Show outdated Hide outdated cabal-install/Distribution/Client/CmdSdist.hs
Show outdated Hide outdated Cabal/doc/nix-local-build.rst
Show outdated Hide outdated Cabal/doc/nix-local-build.rst
@quasicomputational

This comment has been minimized.

Show comment
Hide comment
@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

Oh, one more thing: can --list-only grow a -z / --null option, that will separate file names with a NUL byte rather than a newline, so that anything processing the file list downstream doesn't break when files have newlines in their name?

Collaborator

quasicomputational commented Jun 20, 2018

Oh, one more thing: can --list-only grow a -z / --null option, that will separate file names with a NUL byte rather than a newline, so that anything processing the file list downstream doesn't break when files have newlines in their name?

@quasicomputational

This comment has been minimized.

Show comment
Hide comment
@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

Ah, found a bug: cabal new-sdist Cabal --output-dir . drops the tarball in Cabal/. That should be relative to the CWD of the invocation. Same goes for cabal new-sdist Cabal --output-dir foo, where it'll try to put it in Cabal/foo, rather than ./foo

Collaborator

quasicomputational commented Jun 20, 2018

Ah, found a bug: cabal new-sdist Cabal --output-dir . drops the tarball in Cabal/. That should be relative to the CWD of the invocation. Same goes for cabal new-sdist Cabal --output-dir foo, where it'll try to put it in Cabal/foo, rather than ./foo

@quasicomputational

This comment has been minimized.

Show comment
Hide comment
@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

Sorry, I spotted another thing on second look: cabal new-sdist Cabal --list-only -o - does not work as advertised in the docs: it tries to write to -/Cabal-2.3.0.0.list, rather than stdout. I'm also not sure that -o - is sensible for generating multiple tarballs; maybe it should error out in that case?

Collaborator

quasicomputational commented Jun 20, 2018

Sorry, I spotted another thing on second look: cabal new-sdist Cabal --list-only -o - does not work as advertised in the docs: it tries to write to -/Cabal-2.3.0.0.list, rather than stdout. I'm also not sure that -o - is sensible for generating multiple tarballs; maybe it should error out in that case?

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 20, 2018

Collaborator

I forgot to make the base path absolute before we start hopping around, that's trivial to fix and... Oh wow I have no idea how I managed to not catch that.

Collaborator

typedrat commented Jun 20, 2018

I forgot to make the base path absolute before we start hopping around, that's trivial to fix and... Oh wow I have no idea how I managed to not catch that.

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 20, 2018

Collaborator

I will write more tests later today.

Collaborator

typedrat commented Jun 20, 2018

I will write more tests later today.

@quasicomputational

This comment has been minimized.

Show comment
Hide comment
@quasicomputational

quasicomputational Jun 20, 2018

Collaborator

I can chip in with a bunch as well. Thanks for the changes so far; I think that tests and a little bit more documenting (which I'll prepare a PR into your branch for) and then it's in great shape in terms of nuts and bolts.

Collaborator

quasicomputational commented Jun 20, 2018

I can chip in with a bunch as well. Thanks for the changes so far; I think that tests and a little bit more documenting (which I'll prepare a PR into your branch for) and then it's in great shape in terms of nuts and bolts.

quasicomputational added some commits Jun 20, 2018

Some tests for new-sdist
They're not all passing, but I think that that's a bug in the code
rather than the tests.
@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 20, 2018

Collaborator

I gotta say, huge huge props to @quasicomputational for how much help they've been today.

Collaborator

typedrat commented Jun 20, 2018

I gotta say, huge huge props to @quasicomputational for how much help they've been today.

@typedrat

This comment has been minimized.

Show comment
Hide comment
@typedrat

typedrat Jun 21, 2018

Collaborator

The Appveyor failure is specious, it's ready to merge unless there is a substantial yet heretofore unvoiced objection.

Collaborator

typedrat commented Jun 21, 2018

The Appveyor failure is specious, it's ready to merge unless there is a substantial yet heretofore unvoiced objection.

@quasicomputational

This comment has been minimized.

Show comment
Hide comment
@quasicomputational

quasicomputational Jun 21, 2018

Collaborator

The AppVeyor failure is actually half-legit: it's testing for a path with slashes in the .list file, but of course it's backslashes on Windows. I managed to make that exact same mistake a few weeks ago and apparently didn't learn from it the first time. typedrat#5 fixes that and makes a few other tweaks to the tests; with that merged I'm happy with the test coverage and the code itself.

Collaborator

quasicomputational commented Jun 21, 2018

The AppVeyor failure is actually half-legit: it's testing for a path with slashes in the .list file, but of course it's backslashes on Windows. I managed to make that exact same mistake a few weeks ago and apparently didn't learn from it the first time. typedrat#5 fixes that and makes a few other tweaks to the tests; with that merged I'm happy with the test coverage and the code itself.

quasicomputational added some commits Jun 21, 2018

Test that new-sdist gets globbing right
The code does the right thing, but it's slightly fragile since it
depends on directory switching, so make sure it stays right.
Make sure that all failure paths in new-sdist are tested
This one wasn't actually testing what it said it was testing. Oops.

@quasicomputational quasicomputational referenced this pull request Jun 22, 2018

Merged

Applying flag normalisation to local packages. #5287

4 of 4 tasks complete
@23Skidoo

LGTM modulo minor comments.

``cabal new-sdist [FLAGS] [TARGETS]`` takes the crucial files needed to build ``TARGETS``
and puts them into an archive format ready for upload to Hackage. These archives are stable
and two archives of the same format built from the same source will hash to the same value.

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

For autogen-modules we can't guarantee this 100% since we have no control over non-determinism in preprocessors, especially custom ones. For example, a preprocessor can add a comment with the date when the file was generated, or have output that depends on system locale settings. I guess that we can settle on saying that the behaviour is undefined when a preprocessor is non-deterministic in a way that can be observed from compiled code.

@23Skidoo

23Skidoo Jun 24, 2018

Member

For autogen-modules we can't guarantee this 100% since we have no control over non-determinism in preprocessors, especially custom ones. For example, a preprocessor can add a comment with the date when the file was generated, or have output that depends on system locale settings. I guess that we can settle on saying that the behaviour is undefined when a preprocessor is non-deterministic in a way that can be observed from compiled code.

This comment has been minimized.

@typedrat

typedrat Jun 24, 2018

Collaborator

autogen-modules shouldn't (and it's a bug if they are, that probably needs a test to be sure) be included for this very reason.

@typedrat

typedrat Jun 24, 2018

Collaborator

autogen-modules shouldn't (and it's a bug if they are, that probably needs a test to be sure) be included for this very reason.

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

You're correct! Sorry, it's getting a bit late here.

@23Skidoo

23Skidoo Jun 24, 2018

Member

You're correct! Sorry, it's getting a bit late here.

- ``-z``, ``--null``: Only used with ``--list-only``. Separates filenames with a NUL
byte instead of newlines.
``new-sdist`` is inherently incompatible with sdist hooks, not due to implementation but due

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

OK from me, let's deprecate them.

@23Skidoo

23Skidoo Jun 24, 2018

Member

OK from me, let's deprecate them.

@@ -1,6 +1,8 @@
-*-change-log-*-
2.4.0.0 (current development version)
* Add 'new-sdist' command (#5389). Creates stable archives based on

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

Please also add a note to the Cabal changelog saying that sdist hooks are now deprecated and autogen-modules and build-tool-depends should be used instead.

@23Skidoo

23Skidoo Jun 24, 2018

Member

Please also add a note to the Cabal changelog saying that sdist hooks are now deprecated and autogen-modules and build-tool-depends should be used instead.

Show outdated Hide outdated Cabal/Distribution/Simple/UserHooks.hs
Show outdated Hide outdated cabal-install/Distribution/Client/CmdSdist.hs
sdistAction :: SdistFlags -> [String] -> GlobalFlags -> IO ()
sdistAction SdistFlags{..} targetStrings globalFlags = do
let verbosity = fromFlagOrDefault normal sdistVerbosity

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

If you use Emacs, you can make formatting prettier here (and elsewhere) with M-x align-regexp =. If you don't, don't bother.

@23Skidoo

23Skidoo Jun 24, 2018

Member

If you use Emacs, you can make formatting prettier here (and elsewhere) with M-x align-regexp =. If you don't, don't bother.

| listSources -> "-"
| otherwise -> distSdistFile distLayout (packageId pkg) archiveFormat
createDirectoryIfMissing True (distSdistDirectory distLayout)

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

Strictly speaking, this is not needed when outputting to stdout, but no-one probably cares.

@23Skidoo

23Skidoo Jun 24, 2018

Member

Strictly speaking, this is not needed when outputting to stdout, but no-one probably cares.

| otherwise ->
mapM_ (\pkg -> packageToSdist verbosity (distProjectRootDirectory distLayout) format (outputPath pkg) pkg) pkgs
data IsExec = Exec | NoExec

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

No boolean blindness, nice!

@23Skidoo

23Skidoo Jun 24, 2018

Member

No boolean blindness, nice!

getPkg pid = case find ((== pid) . packageId) pkgs' of
Just pkg -> Right pkg
Nothing -> error "The impossible happened: we have a reference to a local package that isn't in localPackages."

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

Perhaps add a TargetProblemImpossible and use it instead of error here?

@23Skidoo

23Skidoo Jun 24, 2018

Member

Perhaps add a TargetProblemImpossible and use it instead of error here?

This comment has been minimized.

@typedrat

typedrat Jun 24, 2018

Collaborator

Didn't do it since it's harder to get rid of the one in flatten and I don't feel like it's worth the work when there's another five lines above.

@typedrat

typedrat Jun 24, 2018

Collaborator

Didn't do it since it's harder to get rid of the one in flatten and I don't feel like it's worth the work when there's another five lines above.

listPackageSources verbosity (flattenPackageDescription $ packageDescription pkg) knownSuffixHandlers
let write = if outputFile == "-"
then putStr . withOutputMarker verbosity . BSL.unpack

This comment has been minimized.

@23Skidoo

23Skidoo Jun 24, 2018

Member

Maybe add a helper function for this to D.S.Utils?

@23Skidoo

23Skidoo Jun 24, 2018

Member

Maybe add a helper function for this to D.S.Utils?

typedrat added some commits Jun 25, 2018

@typedrat typedrat merged commit 4bdf54d into haskell:master Jun 25, 2018

0 of 2 checks passed

continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@typedrat typedrat moved this from In Progress to Done in Have new-build become build (GSOC2018) Jun 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment