Add target to upload to CTAN

[josephwright]

Similar to ctanupload, it would be useful if something like l3build upload <meta-data> could submit a package directly to CTAN.

[josephwright]

I've asked the CTAN team about an API for this: one is in development (https://github.com/ge-ne/ctan-o-mat) but is not yet production-ready. On the basis that the upload form is not a recommended target (not 'stable'), we'll wait and see what happens about an API.

[wspr]

Sensible — the last thing we want is to develop against a moving target.

[ge-ne]

The API is in place and the client (ctan-o-mat) can be used to validate and upload packages.

[josephwright]

Cool: I'll perhaps look at this shortly.

[wspr]

I took a brief look; ctan-o-mat itself uses Perl; I couldn't find a reference for the CTAN API but even if we went that route I guess we'd need to use a Curl library for Lua.

@ge-ne — Does ctan-o-mat work out of the box on all platforms in all distributions?

Perhaps a good starting point would be to have l3build just generate the .pkg file that ctan-o-mat uses, then it's up to the user to manually invoke ctan-o-mat after checking all the details are good.

[eg9]

On 30 Nov 2017, at 00:33 , Will Robertson ***@***.***> wrote: I took a brief look; ctan-o-mat <https://github.com/ge-ne/ctan-o-mat> itself uses Perl; I couldn't find a reference for the CTAN API but even if we went that route I guess we'd need to use a Curl library for Lua. @ge-ne <https://github.com/ge-ne> — Does ctan-o-mat work out of the box on all platforms in all distributions? Perhaps a good starting point would be to have l3build just generate the .pkg file that ctan-o-mat uses, then it's up to the user to manually invoke ctan-o-mat after checking all the details are good.

I don’t think Windows has Perl by default. Not by direct knowledge, of course. ;-) http://editrocket.com/articles/perl_windows.html Ciao Enrico

[davidcarlisle]

On 29 November 2017 at 23:33, Will Robertson ***@***.***> wrote: I took a brief look; ctan-o-mat <https://github.com/ge-ne/ctan-o-mat> itself uses Perl; I couldn't find a reference for the CTAN API but even if we went that route I guess we'd need to use a Curl library for Lua

luatex is compiled with lua socket library so in principle I think you should be able to make the connection directly from lua without needing curl etc but not actually tried this .

…

@ge-ne <https://github.com/ge-ne> — Does ctan-o-mat work out of the box on all platforms in all distributions? Perhaps a good starting point would be to have l3build just generate the .pkg file that ctan-o-mat uses, then it's up to the user to manually invoke ctan-o-mat after checking all the details are good. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNcAndIwsuYm4c0kswJbhjXDsb7gcZzks5s7em3gaJpZM4OEjTs> .

[wspr]

Ah, thanks — I was looking at the wrong place in the luatex manual. Now we just need to get `texdoc luasocket` to produce something useful :)

[davidcarlisle]

This seems to work although it just hooks the post data to stdout... but it reports a valid connection to a test server.

…

-- load the http module local http = require("socket.http") local ltn12 = require("ltn12") r, c, h = http.request { method = "POST", url = "http://httpbin.org/post?abc=123&xyz=hello", sink = ltn12.sink.file(io.stdout) } print('r: ' .. r) print('c: ' .. c) for k,v in pairs(h) do print(k .. ': ' .. v) end

On 30 November 2017 at 00:03, Will Robertson ***@***.***> wrote: Ah, thanks — I was looking at the wrong place in the luatex manual. Now we just need to get `texdoc luasocket` to produce something useful :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNcAnzyUpYBnkVecnPFI8jr9ik4jVNEks5s7fDqgaJpZM4OEjTs> .

[ge-ne]

On 30 November 2017 at 00:33 Will Robertson ***@***.***> wrote: I took a brief look; ctan-o-mat https://github.com/ge-ne/ctan-o-mat itself uses Perl; I couldn't find a reference for the CTAN API but even if we went that route I guess we'd need to use a Curl library for Lua.

You can find a description of the API under https://ctan.org/help/submit I am currently reworking the upload form which might result in slight improvements of the API. Afterwards it will be linked in the help pages of CTAN.

@ge-ne https://github.com/ge-ne — Does ctan-o-mat work out of the box on all platforms in all distributions?

Well, Perl is available on all platforms. I have minimized the requirements and use only the rather stable LWP library. I can't tell you anything about _all_ operating systems and variants. But Perl and LWP are usually availanle on UNIX-like systems. This includes Windows with Cygwin -- but not plain Windows alone.

Perhaps a good starting point would be to have l3build just generate the .pkg file that ctan-o-mat uses, then it's up to the user to manually invoke ctan-o-mat after checking all the details are good.

Ciao Gerd

…

—You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub #1 (comment) , or mute the thread https://github.com/notifications/unsubscribe-auth/ACPQTPYLbl5lg6A5ovyxFOK7mV89uVnTks5s7em4gaJpZM4OEjTs .

[cereda]

I will take a look later on. 😉

[davidcarlisle]

#1 (comment)

It turns out that lua socket accepts https URL syntactically but it always uses the http protocol as it has no ssl support, so is rejected by the ctan upload. You could in principle use the luasec ssl module which has a compatible ssl aware interface: Philipp Gesang has an example hooking ssl support to luatex here
https://tex.stackexchange.com/a/131947
but the lua ssl module isn't bundled with luatex or even, as far as I can see, in the stock lua 5.2 I have from cygwin, so it would mean a dependency to install this, so it might be simpler to use curl (context seems to assume curl is available) or to use ctan-o-mat which is definitely available, so an l3 function that just built the ctan-o-mat config and called ctan-o-mat might be best. Unless I am missing some way of getting https functionality from texlua.

[josephwright]

@davidcarlisle I think that came up recently on the LuaTeX list (in some other context): you are likely right.

[wspr]

Thanks David. A ctan-o-mat interface is certainly a good stop-gap for the foreseeable future and reduces some development effort needed.

[davidcarlisle]

I attach a lua wrapper for ctan-o-mat (ctan-post.lua) and a sample build.lua calling it, the same input form could be use to generate a curl (or whatever) https post body rather than using ctan-o-mat if it turns out to be more convenient to assume curl is in the path than ctan-o-mat/perl

(.txt extension not .lua for github attachment restrictions)
build.txt
ctan-post.txt

[davidcarlisle]

I notice that since April curl is available by default in windows, so curl is becoming a more viable default back end, I have extended the idea in the above comment so it works with ctan-o-mat or curl.

To allow experimentation with the details before committing to integration in l3build I checked it in to a repo here

https://github.com/davidcarlisle/ctan-post

I don't plan to release it as a self standing tool but it seems to be working well enough to check in somewhere.

[ge-ne]

I have had a look at your implementation. I have seen one feature which you might want to reconsider.

ctan-o-mat does not know a lot about the argments it collects and submits. The description is retrieved from a service on CTAN. Thus it is possible to cope with modifications of the data model on CTAN in the future. For instance if attributes are added or deleted or the length of fields is adjusted.

You have hardcoded these values into your script. Maybe you want to query the same API to avoid the hardcoding...

[davidcarlisle]

@ge-ne yes thanks, I wondered about that, but I went for a fixed list initially, partly to test getting something working. Actually I had some problems with the list API, but I need to return to that, it may have been user error.

The version checked in at https://github.com/davidcarlisle/ctan-post isn't really intended as a user-facing command, just experiments with getting the https post code active.

However even if we do finally include this in l3build I suspect that dynamically querying the list of fields may not be the best. Even if it works at the code level, you need to document to the user the possibilities in the Lua interface and so having a fixed interface has advantages. I tried to make it so that adding a field is just one extra line of code so tracking changes at the server end should not be hard.

I don't know, it's certainly something we should come back to when integrating this in to l3build.

While you are here... :-)

Is there an easy way to check if newlines etc in multi-line fields reach you correctly? Using the curl interface there is some variability in that the '-delimited multiline string is being passed to a curl --form parameter but in the system specific shell.

It passes validation with your .validate URL but I don't think I can check what the actual text looks like without submitting a package and seeing the return email?

David

[ge-ne]

I have two ways. Finally I have tested against my dev instance of the CTAN portal, That's the advantage of being on both sides;-)

To start with I have written a small CGI script which simply writes it's body to a file. Since the portal does not alter whitespace in any way you can see what is transmitted to the server. I am using a Perl script but you are free to use any language you like. I can provide my Perl script...

[davidcarlisle]

@ge-ne I'd wondered about setting up a test server for checking the post body, if you could share your perl code that would give me a head start thanks.

[ge-ne]

I have an Apache running anyhow. I have configured it to execute CGI scripts anywhere under documentroot. Then I have placed the following script as show.cgi (and made it executable):

#!/usr/bin/perl -W

print <<__EOF__;
Content-type: text/plain

__EOF__

while (<>) {
    print $_;
}

This script simply returns the body as plain text. Replacing print by print STDERR writes it to the log file.

[davidcarlisle]

ah thanks! I was thinking that you needed to extract the params, but of course not, thanks I have an apache server with cgi enabled so I should manage to drop that in,

[davidcarlisle]

an initial sketch integration into l3build has been added in the ctan-post branch

[wspr]

@ge-ne — I notice that CTAN packages list an explicit copyright string (e.g., for https://ctan.org/pkg/addlines it is "2007–2018 Will Robert­son")

Is it intentional that this isn't included as part of the API? (No problem if CTAN automatically generates such strings, just checking it's not extracted manually from the package licence.)