FAQ: Creating your first package

[ma-laforge]

No description provided.

[ViralBShah]

This looks great. Just wondering aloud if we should use this somewhere in the manual itself, instead of in the FAQ. And then have a shorter FAQ entry that links to the appropriate places in the manual?

[ma-laforge]

That sounds reasonable. How about in the "modules" section? It appears to have changed significantly in v1.7 docs:

https://docs.julialang.org/en/v1.7-dev/manual/modules/

Also: Would it not be better to call it something like "modules and packages"?

[ViralBShah]

Thinking out aloud, perhaps the Pkg.jl documentation the right place. Perhaps that should be improved and the FAQ entry can then point there?

https://julialang.github.io/Pkg.jl/v1/creating-packages/

There is the issue of how should people discover the Pkg documentation in the first place.

[ma-laforge]

That seems reasonable to me.

I also agree that for this information to be useful: we need to consider the issue on how people should discover the documentation.

(I myself was unaware there was more info about packages in the Pkg.jl documentation.)

Let me know.

[ViralBShah]

Let's wait for @KristofferC, @StefanKarpinski or others to chime in.

[fredrikekre]

There is the issue of how should people discover the Pkg documentation in the first place.

It is linked to in https://docs.julialang.org/en/v1/stdlib/Pkg/ but perhaps good to link to it in other places too.

[vtjnash]

I agree with putting this in Pkg instead, but think we should keep the FAQ entry here too to link to it, with lots of keywords:

### Creating your first Julia package

For package management questions, including Pkg versions, `[compat]` entries, JuliaProject.toml, JuliaManifest.toml, see
<https://docs.julialang.org/en/v1/stdlib/Pkg/>.

[ViralBShah]

I feel this is better to merge than to not. Any concerns if we merge as is?

[ViralBShah]

Had to make a small edit to trigger the whitespace check.

[ViralBShah]

This probably needs rebasing to master.

[timholy]

Personally, I think PkgTemplates has many advantages (e.g., all packages should have CI & TagBot, and many should have Documenter documentation, but setting up either manually is a pain). I'd suggest we should direct most people to use it.

[ViralBShah]

Good point. Maybe we just update this FAQ entry to point people to PkgTemplates?

[ma-laforge]

Coherent example with PkgTemplates?

@timholy : I'm definitely fine with putting more emphasis on the PkgTemplates flow, but I'd personally like to see a walkthrough-type of introduction instead of just a nuts-and-bolts-type introduction (what I see on https://invenia.github.io/PkgTemplates.jl/stable/).

Does such a coherent walkthrough exist? I have not found any.

Issues with this PR (retrospect)

Having read more posts on discourse, I think this PR is both too heavy and not detailed enough. Getting clarity on this subject likely requires better structuring/examples/etc...

Maybe a "Tutorial" section?

Maybe what the docs need is a "tutorial" section on creating packages. That might make it easier for new users to locate the information. ATM: I find this info difficult to find in the Julia docs TOC.

Some issues to tackle in a more coherent/walkthrough fashion:

• How to add custom (local) packages with ]add.
• Code loading: using/import vs include().
• How modules map onto/into packages.
• Adding my new PkgTemplates-created package to 2 projects (Introduction to Projects/environments).
• What's special about the "default" project/environment (ex: v1.6)?
• Practical use-cases for creating "package libraries" with Base.LOAD_PATH.
• How to build/add light-weight, single-file packages.
• Subtleties of Git-based packages vs non-Git-based (add vs dev)
• Practical use-cases for environment stacks.
• 1 or 2 different ways one can split larger packages into multiple files/multiple modules using include().
• Projects ARE environments (defined by Project.toml).
• Packages HAVE environments (defined by Project.toml... unless added with Base.LOAD_PATH).

Many of these issues could link to other areas in the docs --- but having a central location to find things would be nice.

[timholy]

I see your point. I think it would be best to put tutorial material in PkgTemplates so that it can be doctested. (Tutorials must work, hence they need testing to avoid going stale.) Then direct traffic there from here. CC @nickrobinson251 who was a very responsive maintainer for some recent changes I made to PkgTemplates.

[nickrobinson251]

i'd be very happy to have this kind of thing in PkgTemplates.jl! Although i'm equally happy if it is decided that e.g. the Pkg.jl docs is a more suitable home, and for the list of tutorials suggested by @ma-llaforge think Pkg.jl probably is the best place. I of course think PkgTemplates.jl is a good thing for many people to use for starting a package, so am in favour of it being more discoverable as well as more helpful, especially for newcomers.

(as an aside: I also wonder if we should move PkgTemplates.jl to be under the github.com/JuliaLang/ org to help with discoverability and/or if that makes us more comfortable about linking to PkgTemplates.jl from Pkg.jl docs. I just raise this as an option in case it's something that would make a difference to the decision, but if it doesn't then we can leave this question alone.)

[ma-laforge]

@timholy Though I haven't yet figured out how to do doc-testing myself, it does sound like a good idea. I also find it reasonable to place the tutorials in Pkg.jl (@nickrobinson251).

Time restrictions

At the moment: I will be somewhat unavailable to give this another go.

Nonetheless, I still try to help others on discourse every now and then --- but I'm afraid I often don't quite succeed at finding a balance between being succinct and providing sufficient information.

Previous discussions on package creation

I have been making attempts to describe how to create packages, provide use cases, delineate concepts, etc.

Maybe these attempts might can help get the ball rolling:

High-level overview of modules & packages in an attempt to alleviate confusion:
--> https://discourse.julialang.org/t/could-we-make-first-class-support-for-packages-that-are-just-files-and-not-repositories/36931/20

Somewhat similar (but less well explained in my opinion - read this last - if at all):
--> https://discourse.julialang.org/t/how-to-add-a-folder-path-in-your-jl-script/61901/8

Creating your first package (incl. file organization; similar to this PR)
--> https://discourse.julialang.org/t/how-does-the-module-system-actually-work/53548/3

Tips for include vs import/using
--> https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/3

Tips for directory structure/LOAD_PATH
--> https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/4

Link to more relevant threads:
--> https://discourse.julialang.org/t/proper-way-of-organizing-code-into-subpackages/52835/5

Packages, modules & nomenclature in comparison to C/C++:
--> #37167

Some Issues with my explanations

I started writing that Julia "packages" mapped more closely to the concept of "Software modules" than what Julia calls "module"s. I am no longer of this opinion.

• I now believe that Julia "modules" do indeed provide the base encapsulation necessary for creating "software modules" (with the exception that nothing is truly hidden in Julia).
• I now believe the added-value of Julia "packages" is that they provide a practical/efficient means of distributing/versioning said modules (+submodules).

I'm sure that was obvious to many of you already, but I struggled a fair bit with deciding what to bundle up as a "package", and what simply needs to be a "module".

I think much of my confusion happened because:

• Multiple dispatch sort of eliminates the need for subdividing each module into its own namespace.
• Julia doesn't really do information hiding, so this sort of encapsulation doesn't really exist in Julia.

[ma-laforge]

@nickrobinson251: I would really like to see some sort of tutorial on how to use PkgTemplates myself. I'm not always that imaginative, so I often need examples to get it.

Making it easier to set up one's pacakge for CI, tagbot, & documenter sounds awesome! I finally did get CI to work -- but I've avoided figuring out documenter to limit my growing pains.

[nickrobinson251]

thanks, @ma-laforge -- if you open an issue on PkgTemplates.jl with anything in particular it'd be good to explain/demo more/better that would be really helpful, then i can have a go at improving docs for those specific things. Otherwise, i will take a closer look at this PR and think about how PkgTemplates.jl docs can be improved generally, but might not get to it this month.

[ma-laforge]

FYI:

I (very) briefly tried to look into PkgTemplates.jl. Immediately, I noticed it wanted to know which github username to use (JuliaCI/PkgTemplates.jl#327).

Revision control: a deterrent for new users?

Though I am sure this is a good idea for proper package deployment, I am not convinced the best way for new users to get acquainted with packages and modules is to immediately start thinking about Github accounts.

Ideally, I think the first introduction to Julia packages & modules should be limited to simple file/folder-based structures. It is probably best not to even mention revision control. That can be a show stopper to many people in the engineering/science fields (YMMV).

I know revision control is very important -- and that people should definitely make that effort to understand the tools. That said: having to learn too many things at once can be a deterrent to adopting a new language.

Will Julia make me efficient?

The questions I have noticed on discourse seems to imply that potential adopters coming from other languages have trouble making "generic, re-usable components" that can easily included in multiple projects. Of course, that's not explicitly what they write -- but I'm pretty certain that's what most people are (eventually) seeking when they develop in any language: How can I build a solid, generic toolset in such a way that makes me reach my end goals more efficiently?

Back to PkgTemplates

If @timholy is right: I'm all for putting the spotlight on PkgTemplates to help developers build solid, well tested packages. But something tells me that shouldn't be step #1.