Fix #355 Add support for includes

[mpilgrem]

Follows the pattern of existing install-includes. Fixes:

• Why does hpack not use the includes property? #355

EDIT: A test added, again following install-includes.

[sol]

This looks good, just one question:

https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#pkg-field-includes says that this can be used for "both header files that are already installed on the system and to those coming with the package to be installed".

@mpilgrem for includes that come with the package, did you try if cabal includes those into the tarbal? Or does cabal require you to also list them under extra-source-files?

If it's the latter, then we would need to address this somehow to be consistent with don't repeat yourself.

[mpilgrem]

@sol, as far as I can tell:

• for Cabal's install-includes: <filename list>, sdist will include the files*** in the archive and Cabal will copy the files to $libdir/includes when the package is installed; and
• for Cabal's includes: <filename list>, sdist will not include the files in the archive. If you want one or more of the files to be the archive, you need to list them under Cabal's extra-source-files:.

EDIT3: *** As a complication, sdist only includes the first such file, if the combination of Cabal's include-dirs: and Cabal's install-includes: specifies more than one file of the same name.

So, perhaps Hpack does need two keys:

• one to correspond to Cabal's includes: only (perhaps named includes-system); and
• the other to correspond to Cabal's includes: and Cabal's extra-source-files: used together (perhaps named includes-package),

and, presumably, some form of de-duplication so that a file specified under Hpack's includes-package (if that key name is adopted) and Hpack's extra-source-files does not get rendered twice under Cabal's extra-source-files.

EDIT: To add to the complication, if you want to use the files listed under Cabal's install-includes when compiling the package you have to list then also under Cabal's includes. However, I don't think Hpack distinguishes between:

• Cabal's install-includes that you want to use; and
• Cabal's install-includes that you don't want to use.

EDIT2: The Cabal User Guide also has:

The former files [header files that are already installed on the system] should be found in absolute paths, while the latter files [header files coming with the package to be installed] should be found in paths relative to the top of the source tree or relative to one of the directories listed in include-dirs.

If 'should be' is to be read as 'must', then perhaps Hpack could assume that all absolute file paths are system headers and all relative paths are to be included in extra-source-files.

[mpilgrem]

Rebasing on main branch.

[sol]

The former files [header files that are already installed on the system] should be found in absolute paths, while the latter files [header files coming with the package to be installed] should be found in paths relative to the top of the source tree or relative to one of the directories listed in include-dirs.

If 'should be' is to be read as 'must', then perhaps Hpack could assume that all absolute file paths are system headers and all relative paths are to be included in extra-source-files.

Yes, that's what I was hoping for. However, giving it more thought, I'm not sure if this is a sound assumption. My naïve assumption would be that a relative path actually can refer to a system include depending on what -I options are passed to the C compiler, or even if a corresponding include file is on any system standard directory for include files.

I still imagine that an algorithm could be, for each relative file name listed under include

1. If the file exists relative to package.yaml then add it to extra-source-files
2. For each relative directory listed under include-dirs If the file exists relative to the directory then add it to extra-source-files

If there are multiple hits, we add all of them, and leave resolving of precedence to cabal/gcc. I think this ensures that an in-place build and a cabal install use the same includes.

One downside of this approach is that it's complicated to implement, requiring similar code as what we have for module inference.

[mpilgrem]

@sol, I agree that it is complicated to implement. I am happy to give it a go, but it could take me some time. Could you tolerate a step-by-step approach? That is:

• Step 1: add this naive approach to includes (one that does not automatically populate extra-source-files); and
• Step 2: extend support for includes to automatically populate extra-source-files?

[sol]

Taking a step back, how crucial is this feature?

The Cabal documentation says that they are used for compilations via C. Traditionally, in the context of GHC, "via-C" means unregistered builds / bootstrapping, which is pretty exotic, so I kind of doubt that this is what the Cabal docs refer to.

So what I kind of assume this is actually doing is pass -include to the C compiler when compiling c-sources. This is still pretty exotic though, as usually you would just use #include-statements instead.

Where include could actually be somewhat useful is, if Cabal would also pass them to ghc via -optP-include. Not sure if that's the case, the docs don't say anything about it and I'm currently away from the computers, so I can't try.

[sol]

@sol, I agree that it is complicated to implement.

Actually, I think I was wrong on the "complicated". I think you can just take the total sum of all includes and include-dirs (from all sections and conditional branches) and the proceed as I described above. This may, as a result include more .h files then necessary (as some combinations of includes and include-dirs may never occur together), but again it shouldn't be harmful.

I am happy to give it a go, but it could take me some time.

Great 😊

Could you tolerate a step-by-step approach?

main should always be in a releasable state. So I think better do it as one PR, but again, I don't think it's as complicated as I originally thought.

[mpilgrem]

@sol, I had been assuming - perhaps incorrectly - that it was needed whenever a Haskell package used C code. My 'template' had been the Cabal file for Win32, eg https://hackage.haskell.org/package/Win32-2.14.0.0/Win32.cabal. Perhaps there is something 'special' about the Win32 package. EDIT: I'll see if I can improve my understanding.

[sol]

@sol, I had been assuming - perhaps incorrectly - that it was needed whenever a Haskell package used C code.

Definitely not needed, see e.g. https://github.com/sol/simdutf-haskell/blob/main/package.yaml#L15

My 'template' had been the Cabal file for Win32, eg https://hackage.haskell.org/package/Win32-2.14.0.0/Win32.cabal. Perhaps there is something 'special' about the Win32 package.

I am gonna look at it

[mpilgrem]

Here is another example of a package using includes:: https://hackage.haskell.org/package/mintty-0.1.4/mintty.cabal

[mpilgrem]

I have seen something that implied to me that includes: may be required when header files are not co-located with c-sources: etc.

[sol]

I have seen something that implied to me that includes: may be required when header files are not co-located with c-sources: etc.

I don't think so, that is what include-dirs is for.

[sol]

Looking at ghc/packages-Win32@2a56e45#diff-851be7da3fa86b5b2d90284d7440f3d15f9749698c2c14d02013641124050943R100

It's not a apparent to me what the benefit of includes could possibly be.

[sol]

Here is another example of a package using includes:: https://hackage.haskell.org/package/mintty-0.1.4/mintty.cabal

Again, it's not a parent to me what includes would achieve here, given: https://github.com/RyanGlScott/mintty/blob/master/src/System/Console/MinTTY/Win32.hsc#L57-L59.

@RyanGlScott do you know for reason includes would be necessary here?

[mpilgrem]

EDIT: I am going to further edit this post as I flesh out my understanding of the history. What I understand so far:

• The original Cabal specification includes includes:, with the note not yet used: `https://downloads.haskell.org/cabal/Cabal-1.0/doc/pkg-spec.pdf.

• Before GHC 6.4, packages did not have versions. GHC 6.4, released on 11 March 2005, introduced package versions and Cabal-1.0. GHC 6.4 also introduced GHC's INCLUDE pragma, which was not part of the Haskell 98 report.

• The Cabal-1.0 guide did not document an extra-source-files field and did not have a working Setup.hs sdist command. It documented:

  includes: filename list
  A list of header files from standard include directories or those listed in include-dirs, to be included in any
  compilations via C. These files typically contain function prototypes for foreign imports used by the package.
  
  include-dirs: directory list
  A list of directories to search for header files, both when using a C preprocessor and when compiling via C.
  
  c-sources: filename list
  A list of C source files to be compiled and linked with the Haskell files.
  
  If you use this field, you should also name the C files in CFILES pragmas in the Haskell source files that use
  them, e.g.:
  
      {-# CFILES dir/file1.c dir/file2.c #-}
  
  These are ignored by the compilers, but needed by Hugs.
  

• GHC 6.4 came with Win32-1.0. Win32.cabal included:

  include-dirs:   include
  includes:       "HsWin32.h", "HsGDI.h", "WndProc.h"

• GHC 6.4 also came with network-1.0. network.cabal included:

  include-dirs:   include, ../../ghc/includes
  includes:       "HsNet.h"

• GHC 6.4.2 was released on 19 April 2006. Unusually, it updated the Cabal API to Cabal-1.1.4. This brought: cabal-version:, extra-source-files: and a working Setup.hs sdist command.

• GHC 6.6 was released on 11 October 2006. It updated to Cabal-1.1.6. This brought install-includes: and the documentation for includes: had changed, as follows:

  includes: filename list
  A list of header files already installed on the system (i.e. not part of this package) to be included in any
  compilations via C. These files typically contain function prototypes for foreign imports used by the package.
  
  install-includes: filename list
  A list of header files from this package to be included in any compilations via C. These header files will be
  installed into $(libdir)/includes when the package is installed. Files listed in install-includes: should
  be found in one of the directories listed in include-dirs.
  
  install-includes is typically used to name header files that contain prototypes for foreign imports used in
  Haskell code in this package, for which the C implementations are also provided with the package.
  

• GHC 6.8.1 was released on 3 November 2007. It updated to Cabal-1.2.2.0 (which is on Hackage). It documented:

  The includes field now only specifes which include files are automatically included when compiling the package.
  A new install-includes field determines which include files are also installed.
  

• Cabal-1.2.2.0 had in Distribution.Simple.GHC.ghcOptions:

  ++ [ "-#include \"" ++ inc ++ "\"" | inc <- includes bi ]

  making use of GHC 6.8.1's -#include option, which was an alternative to the (preferred) INCLUDE pragma.

• GHC 6.10.1 was released on 4 November 2008. It documented:

  FFI change: header files are now not used when compiling via C. The -#include flag, the includes field in .cabal
  files, and header files specified in a foreign import declaration all have no effect when compiling Haskell 
  source code.
  

• GHC 6.12.1 was released on 14 December 2009. It documented:

  The -#include flag and INCLUDE pragma are now deprecated and ignored. Since version 6.10.1, GHC has generated its
  own C prototypes for foreign calls, rather than relying on prototypes from C header files.
  

[sol]

From what I have tried, includes is neither passed to ghc (via -optP-include) nor to gcc (via -include).

I guess somebody has to go to the Cabal source and dig up the truth 😅

[RyanGlScott]

do you know for reason includes would be necessary here?

I'm not sure I know enough to say for sure. Ultimately, the code in mintty was adapted from win32, which was already using includes before mintty came along.

[mpilgrem]

@sol, having worked through the history above, the includes: field of Cabal files have been ignored by GHC since GHC 6.10.1 (November 2008). So, I amn going to:

• raise a pull request to update the Cabal User Guide; and
• close this pull request.