When will typst be available for journal submissions?

[zkk960317]

I believe that the number of users of typst will increase significantly, when it becomes available in the submission systems of journal publishers (such as Elsevier). So, is typst official or some other organisation pushing for this now? How much further is there to go?

[alerque]

This seems like it would be up to the Journal publishers, not Typst. Given the pressure for years to improve their submission workflows I doubt this will be easy. I suspect only a large amount of pressure from authors will sway the needle.

[frozolotl]

I don't think journals will accept Typst submissions before it has reached stability and is reasonably feature-complete.

Additionally, I don't think it is wise to push for Typst support before that point, else they will just get tired and trust will be more difficult to build up.

[Enivex]

Most journals don't even accept manuscripts requiring LuaTeX instead of pdfTeX, so I don't imagine it'll happen quickly. It would at least be a few years after typst reaches a stable state. Academic publishing moves slowly.

[HPDell]

A practical way may be to convert the Typst file to a LaTeX file and submit that LaTeX file.

[alex180500]

How can you do so?

[Enivex]

How can you do so?

Pandoc or MiTeX, but both are very lossy.

[HPDell]

How can you do so?

I know it's not very easy. But as the official support of converting Typst to LaTeX has been denied and we still need a huge amount of time to wait for Typst to be mature enough and accepted by publishers, Pandoc is the most practical way. I'm still trying to find a reproducible process.

[kmccurley]

We (iacr.org) recently launched a new journal using a custom LaTeX class that we designed as well as a python compilation system that processes metadata. Authors upload a zip file of LaTeX to our server and it compiles in the cloud to produce the PDF. This allows us to do things like insert the DOI and crossmark link into the PDF, but it also allows us to extract all of the metadata from the author's document in order to automatically register the DOI and produce the HTML page that hosts the PDF. We also have a latex log parser and bibtex log parser that automatically flags things for the copy editor (like a missing DOI in a reference). The system is available as open source but isn't really ready for other journals to use it yet. Our system is designed to make sure that no copy editor has to touch the author LaTeX - we simply tell the author to fix things.

While LaTeX is completely dominant in mathematics and computer science, there is always the possibility that authors would want to use another format like Microsoft Word, PreTeXt, ConTeXt, or Typst. In order for us to do this, we have two requirements:

1. the output would have to look roughly the same as what is produced by LaTeX. That means we have to control the look and feel produced by the authors. We do that in LaTeX by doing things like disabling \bibliographystyle, natbib, and some packages that are inconsistent with our style. We also just omit some packages from our cloud compilation service. If authors violate the style, we throw it back to them and in some cases we have simply declined to publish as the author prepared it. When authors stick to the predefined macros of the class, it's easy for them to produce a compliant document.
2. we need to have a system that is almost completely automated. Our copy editors don't need to understand or rewrite LaTeX, and we wouldn't want them to have Typst knowledge either. A requirement for diamond open access publishing is to drive down the price of publishing, and that means we won't spend time fixing problems produced by authors. We are therefore not interested in partial solutions like pandoc that produce only an approximation to the original input.

Unfortunately when I looked at Typst, it appears that it would be difficult to accomplish what we want. LaTeX encourages authors to write logical structure using semantic commands like \section, and they are supposed to leave decisions about what that generates to the underlying document class. Of course authors can always override this, but that is not how LaTeX was intended to be used. This was also mentioned here. I have not found the equivalent of texdoc clsguide to say how to write a document class for authors to follow. I am also afraid that the type of author who gravitates to Typst is the one who wants to write programs instead of articles. LaTeX programming is hard and authors often shy away from it, which is good for a journal.

The way we solve the problem of capturing metadata from the LaTeX is to have it write to a file that is post-processed by python. Apparently that is not possible in Typst, and the developers seem to be against allowing it. As an alternative, we could have the python parse the Typst file to extract what we want, but a parser for Typst is almost as problematic as a parser for LaTeX. The pubmatter package has some overlap with what we want, but I have not investigated it much.

I get the sense that the Typst community does not understand publishing workflows very much. That's not surprising, and the same can be said for LaTeX authors. Maybe someone should first write a Typst plugin for OJS.

[SeSodesa]

In Typst, you define template functions instead of classes. As a publisher, you would simply provide a journal.typ file that contains a main template function appearance to your users. They would then simply write a file with the following contents, to generate a paper that conforms to your appearance guidelines:

#import "journal.typ"

#show: journal.appearance.with(
    authors: (
        "author1",
        "author2", 
        "author3",
    ),
    title: "Paper title",
    keywords: ("word1", "word2", "word3"),
    // Other required fields here.
)

Document content here.

Also, Typst is based on structural markup like LaTeX is. The markup is just different: instead of the LaTeX markup

\section{Title here}
\subsection{Title here}

for section titles, Typst utilizes

= Title here
== Title here

to do the same thing. As one can see, the markup in Typst in simpler than in LaTeX, and that is why I use it. Not because I want to program in Typst.

[jbirnick]

Maybe typst query is what you are looking for to extract the metadata.

[DorianBourgeoisat]

I think you are approaching typst as a Latex user, which is understandable, but it makes you miss the mark, I think.
In typst, there's no "class" yes, but that's because that is a fundamentally latex concept. The typst way of doing it is to pass a document content to a templating function, in another typst file. The "real" document is then mainly made of markup typst, with all the "code" typst in the template file, with show rules calling functions on certain elements, etc ...

Yes you'll often see in the community talks about how to code things, but people aren't coding in their documents, they are coding for their templates and packages, because typst makes it very accessible to beginners and those used to regular programming languages, while I know that only latex wizards dare to write their own classes and packages. People don't talk about the markup syntax, because it's very much like markdown, and it's so easy to learn there's not much need for help.

And yes, you cannot generate an arbitrary file to ingest it in python, but you've got to admit that it's kind of hacky and not very neat. As @jbirnick said, you use typst query (https://typst.app/docs/reference/introspection/query/#command-line-queries) to query the typst compiler for such things, which is, I think, much less janky.

[kmccurley]

I now have a clearer understanding of why there is not much progress toward having journals accept typst. At some point we may return to this question if an author asks for it and is willing to develop the template in a way that is faithful to the style of the journal.

[CGMossa]

What exactly warranted such a reply? You got some very excellent replies to your stream of thoughts. I personally read these messages multiple times, and was very happy with how cordial and accurate the responses were.

[rikhuijzer]

I now have a clearer understanding of why there is not much progress toward having journals accept typst.

If your understanding is "we as a journals have a government-sponsored monopoly so we can do whatever we want no matter how ineffective for authors", then your understanding is correct.

[stephengaito]

@kmccurley Firstly, I must complement you on the detail you have taken describing your open source journal workflow system.

However, I strongly suggest, Typst has been designed for exactly your type of workflow.

To fully understand Typst you need to realise that it is two systems. Firstly it is the typesetting engine typst, but this (open source) typesetting engine has been designed to support Typst as an online (nearly) instant collaboration system.

Typst as an online collaboration system must be first and foremost secure against random authors attempting to break the system.

In your description, you point out that compiling LaTeX from unknown authors is a security risk. To quote from the Checkoway, Shacham and Rescorla article which you reference:

"Nevertheless, it [TeX] allows macros to read and write
arbitrary files. This single capability turns out to
be enough to allow a carefully crafted document
to completely escape TeX’s sandbox."

This is precisely what Typst (as a typesetting engine) forbids in order to support the security of Typst (as an online collaboration tool).

However, Typst (as a typesetting engine) does allow nearly arbitrary metadata to be exported from any typst document. To do this you use the typst query command. The resulting metadata is "exported" to stdout in either JSON or YAML format. This means no random file is written directly to the file system. The only output is placed where the system architect (you) want it and not where a nefarious author might like to put it. Equally, the metatdata output is highly structured (either JSON or YAML), further inhibiting a successful security attack.

While there will always (briefly) be security attacks against JSON (or YAML) as parsed by a specific system programming tool (such as Python), these attack windows will probably only be brief before they are noticed and corrected.

This does not mean that you could/should stop using docker (or better rootless podman) containers to typeset your documents. But by working with Typst's query system, you are seriously reducing your (known) security risks.

Now Typst as a nearly instantaneous collaboration system, makes one further requirement on Typst as a typesetting engine. To be nearly instantaneous, typst is a Pure Functional language (such as Haskell).

This means that when you compute your metadata from an author's document, you must do so inside a Typst "Context", since your metadata is essentially a "side-effect" of the document and not part of the result.

I am at the moment trying to provide a Gerby like tool in which to write out some complex mathematical arguments in mathematical physics. As such I have had to sketch, in the following typst code, collection of my required metadata from my collection of typst documents. My currently very crude code to do this in a Typst template file, is:

#let lpitDocument(
  docId,
  shortTitle: [],
  longTitle: []
) = {
  context {
    let theData = (:)

    let headingData = ()
    for aHeading in query(heading) {
      let loc = aHeading.location()
      let aPageNum = loc.page()
      headingData.push(("theHeading": aHeading, "thePage": aPageNum))

    theData.headings = headingData

    theData.docId = docId
    theData.shortTitle = shortTitle
    theData.longTitle = longTitle
    theData.abstract = query(<abstract>)
    theData.inputs   = sys.inputs
    }
    [ #metadata(theData) <lpitMetaData> ]
  }
  [ #longTitle #label("title") ]
}

Which would be collected using the command line (in a python formatted string):

  f"typst query {docFileName} \"<lpitMetaData>\""

The result of this command line would be to obtain a JSON formatted string on the command's standard out.

Note the "<lpitMetaData>" in the command line is a query for the reference <lpitMetadata> which references the result of the context code block (above).

To use this, an author (me), would place the following lines at the top of their document:

#import "@local/lpit-publisher-papers:0.1.0" : * 

#lpitDocument("fp-diag",
 shortTitle: [ Finger Pieces : Diagrams ],
 longTitle: [ Finger Pieces : A miscellanea of diagrams ])

Of course your system's Typst template would be slightly different and collect different data.

I hope that, as the typst typesetting engine matures, you and other journal systems architects will take a longer and more careful look at Typst.

(I have used LaTeX for many decades, but I see Typst as a very solid future for mathematical typesetting).

[stephengaito]

@kmccurley, In this my second reply to your comments, I would like to
re-raise, more explicitly, your other but very important consideration. I
quote you:

"That means we have to control the look and feel produced by the authors.
We do that in LaTeX by doing things like disabling \bibliographystyle,
natbib, and some packages that are inconsistent with our style. We also
just omit some packages from our cloud compilation service."

In a typst based workflow you would probably similarly limit the included
packages. I suspect you would do this by disabling the automatic download
of packages and providing only the packages you accept, exactly as you
have done for LaTeX.

However, I suspect you are equally worried about how to disable authors
from providing their own typst code to micro-manage the document layout
from inside their document, in ways which break your journal's style.

From the point of view of a systems architect of a journal's automated
publishing workflow, detecting or forbidding such micro-managed style
infractions is critically important.

So as a question to the greater Typst community, how could such
micro-management be detected or foridden?

[stephengaito]

SO here is a crude but very un-typst suggested answer to my question.

Such micro-management of style can be done in any LaTeX document almost as easily as in Typst.

You suggest you detect or forbid such micro-management by actively disabling some LaTeX commands.

While you might be able to do the same in typst, an equally useful way might be to pre-process an author's document using, for example, a simple Python script which looks for instances of the strings #set or #show.

You would not need to understand the content of the document only that it does/does-not contain these strings. If found, that document could simply be rejected in its current form.

Equally, your script could also look for instances of #import and #include and check that the files to be imported are "allowed" (as either they are part of the submission OR they are allowed packages). Again, violations of your rules would result in the paper being rejected in its current form.

While this is an admitted crude way to do this, it is potentially simpler than trying to guess all of the LaTeX commands which might violate your style.

[jbirnick]

I was also thinking about this. From a user perspective:

1. You could modify the compiler to only accept certain show and set rules. (And yes please modify the compiler, don't just search for strings, because detection of the latter can be easily circumvented.)
2. Typst could directly provide a way to prohibit such stuff, either:
   a. via command line options
   b. perhaps more elegantly: it would be great if the template of your journal could somehow say directly in typst syntax that "i forbid any other set rules"

From a compiler perspective:
Technically, I guess one easy and effective way to attack it via Spans, i.e. the location of the source code that caused a show/set rule. One could simply say: "set and show rules are allowed only from packages x,y, and z"

All of that doesn't stop people from directly using arguments to functions though, without any set rules. So one would also need to block that somehow.

[Enivex]

Such micro-management of style can be done in any LaTeX document almost as easily as in Typst.

It's inarguably much easier in LaTeX than in typst

[DorianBourgeoisat]

More elegantly, I think, would be a compiler mechanism to make a show rule override show rules deeper in scope, for example a #show! heading: it => { ... } one could also imagine having multiple "layers" #show! #show!! etc ...

[kmccurley]

My previous comment about understanding the problems came about as a result of reading a lot and trying it out. I started going through the list of issues and found about 5 show-stoppers for us like this and this and this before I stopped reading. I didn't want to expand this into a list of "must-haves" for a journal to accept it. That list might vary from one discipline to another and from one publisher to another. It also appears that github discussions for typst are being moved to a discourse server anyway. The typst team has accomplished a lot, but they already have a lot on their plate. I'll let them establish their own priorities.

We are only one tiny publisher, for a society of only 3000 members. We don't really have the resources to devote to this, and frankly I think we have bigger problems to solve in our publishing pipeline and nobody has asked for typst yet. The academic publishing world is mostly controlled by five or six big commercial publishers (e.g., Springer/Nature, Elsevier, Taylor and Francis, Sage, etc). The large professional societies also run their own publishing branches but some of that is outsourced (e.g., ACM, AMS, IEEE, etc). IACR also outsources most publishing to Springer, and we have done that since the 80s. Most publishers offer almost no visibility into how their workflows work, but I think that LaTeX is much more common than people realize in STEM publishing pipelines. It's what authors write in, and publishers are pretty much saddled with it as their input format. In disciplines outside of STEM, the standard document format is Microsoft Word (OOXML under the covers). If you think LaTeX is bad, you should try reading the 5000 page definition of that format.

Most of the large commercial publishers use large amounts of human labor to transform the author-supplied LaTeX (or OOXML) into another format such as JATS, which can then easily be used to generate PDF or HTML or ePub or whatever. They might use a rough tool like pandoc to get them started, but then they tune it manually to accomplish their typesetting goals. That can be justified when a publisher like Springer charges $3000-6000 to publish an article as open access (ACM is currently charging $1300-1800). Academic publishing is currently under intense pressure from open access mandates, so the economic apple cart that justifies human effort to convert documents may not survive.

We set a goal of establishing a "no-touch" low-cost publishing workflow where we take the author's input and produce the final output automatically. In the process we had to give up a few things like full HTML output (arXiv now has an experimental system that generates HTML from LaTeX, but it fails the "no-touch" requirement). We parse the LaTeX and BibTeX log output to help in the copy editing process. I think I can see a way that we would accomplish pretty much the same thing with typst. The issues with the typst format may eventually get ironed out, and that's for the typst community to work out. Academic STEM publishing may not even end up being that important for the typst ecosystem. It might be useful to have a "publishing" label on issues so that you can accumulate more information on what publishers will want.

I mentioned the possibility of providing a plugin for OJS, and I think that's a potentially good project for someone who can tolerate PHP. OJS is used for the "long tail" of publishing (not the big publishers). I think at least 20,000 journals use it. The vast majority of those are the humanities and social sciences though, and they expect authors to upload Microsoft Word because that is what authors use. OJS has only minimal support for copy editing, and I think this is one pain point for them. They are currently looking at more features to support JATS translation, and a typst => JATS converter might be interesting. At present JATS supports only TeX and MATHML formats for inline and display mathematics, so a typst => JATS converter would have to deal with that. Moreover, I would caution against cutting too many corners in this - we already have approximate translation tools like pandoc for a lossy conversion from one format to another. OJS is also struggling with their technical debt. It took them 3.5 years to add the capability to store multiple affiliations per author.

Academic publishing is a huge can of worms, and the difficulty of making headway there should not be underestimated. LaTeX is thoroughly entrenched and while it has many detractors, learning LaTeX is generally regarded as one of those necessary steps in becoming a successful academic in STEM. Most people can learn the basics in an afternoon and the support ecosystem is gigantic. As a programming language it is definitely showing its age, but the vast majority of authors should not be writing new LaTeX macros.

[Ni-Ar]

I will never understand certain scientists' fixation on submitting a manuscript as source code and relying on journals to publish it as a PDF. Just send the freaking PDF and that's it. This feels like a first-world problem. Or am I missing something?

[Ni-Ar]

Thanks for the explanation. In my field, it is extremely rare to submit a manuscript in LaTeX to a journal. Microsoft World is the gold standard...

[alerque]

Worser ’n worser! My condolences.

[Ni-Ar]

You no idea...

[kmccurley]

To be a bit more specific, publishers usually create the final PDF with data that they supply (e.g., submission date, acceptance date, publication date, license, DOI, crossmark, volume#, issue#, page numbers, etc.) They also sometimes do copy editing in the process (for example, authors are notoriously bad at crafting their bibliographic references). In the future we should expect that LLMs will have a greater hand in copy editing. Publishers also register a fair amount of metadata with crossref.org at the time of publication. It's a bit more complicated than just producing a PDF and putting it on the web.

Much has been written about the academic publishing business, but the fact is that the commercial publishers have a lock on it because the academic world permitted it to happen. The scholarly societies are only a little less exploitative - ACM charges between $1300 and $1800 to publish an article so they can maintain their Manhattan office suite. There are market reasons why it's hard to break their stranglehold on publishing. At present European research funders are having the greatest impact by requiring that the research they pay for becomes open access at the date of publication. The NSF would never do that because the USA worships at the altar of profit. This is strange since the commercial academic publishers are mostly in Europe.

I've been working on an open source LaTeX publishing pipeline, and there doesn't seem to be any competitor in spite of the fact that LaTeX is ancient. Many technologies have strong open source offerings (e.g., email, editors, browsers, web servers, PDF viewers, etc) but this space seems to have been neglected. The strongest open source offerings for journal publishing are OJS and Janeway, but neither one has any real support for LaTeX. The part of the publishing world that they cover is the "long tail" of less well-known journals. OJS has a lot of technical debt that causes them to be pretty slow moving. I filed an issue in June 2021 to support multiple affiliations for an author, and it still hasn't shipped. I had previously suggested that maybe someone should write a plugin to support a production pipeline for typst inside OJS, but it wouldn't be a fun project and honestly I don't think it would get much usage either.

Outside of STEM fields, Microsoft Word is the lingua franca of publishing. If you really want to understand how screwed those fields are, you should try reading the 5000 page specification for the XML underlying the word format. It's one of the worst designed formats that I have ever seen - it even makes PDF look good.

[SeSodesa]

And maybe a final addition to this is that until accessible well-tagged PDFs become the standard, extracting any kind of information (including plain text) from a PDF file in a reproducible manner is a complete pain in the butt. Just take a plain old PDF file and try opening it in a few different PDF readers to see the differences in output.

It is just infeasible for publishers to try and parse such unstructured data and reliably convert that to all of the different formats (Web pages, linearized PDF) that their journals are published in.