From 5e45fc266f61b54acca0543d3ca838132093aaed Mon Sep 17 00:00:00 2001 From: Markus Demleitner Date: Wed, 16 Feb 2022 08:46:38 +0100 Subject: [PATCH 01/15] Unrelated cleanup: Removing legacy README. --- README | 14 -------------- 1 file changed, 14 deletions(-) delete mode 100644 README diff --git a/README b/README deleted file mode 100644 index f28c48a..0000000 --- a/README +++ /dev/null @@ -1,14 +0,0 @@ - -* 2016 * - -This is now the definitive home for VOSI source documents (VOSI.tex and the -various xsd files). - - - -* previous * - -This project was created by Ray Plante. This should not be considered the -definitive home for VOSI source documents; please consult the IVOA GWS twiki -page, http://www.ivoa.net/cgi-bin/twiki/bin/view/IVOA/IvoaGridAndWebServices. - From bf806803f97654265652ba7e623ae2a3720ba5ff Mon Sep 17 00:00:00 2001 From: Markus Demleitner Date: Wed, 16 Feb 2022 09:20:17 +0100 Subject: [PATCH 02/15] Removing spec and requirement for availability --- VOSI.tex | 285 ++++++++---------------------------------------------- local.bib | 8 ++ 2 files changed, 51 insertions(+), 242 deletions(-) create mode 100644 local.bib diff --git a/VOSI.tex b/VOSI.tex index 944c5f3..e0ef985 100644 --- a/VOSI.tex +++ b/VOSI.tex @@ -4,6 +4,7 @@ \usepackage{listings} \lstloadlanguages{XML} \lstset{flexiblecolumns=true,basicstyle=\footnotesize,tagstyle=\ttfamily,showspaces=false} +\usepackage{todonotes} \title{IVOA Support Interfaces} @@ -68,10 +69,13 @@ \section{Introduction} This document describes a set of common basic functions that VO web services provide in the form of a standard support interface in order to -allow for the effective management of the VO. There are three basic +allow for the effective management of the VO. There are two basic support functions that this document describes: The reporting of -capability metadata, the reporting of service availability metadata, and -the reporting of table metadata (if applicable). +capability metadata and +the reporting of table metadata (if applicable). A previous version of +this document also required an interface for reporting service +availability. As discussed in sect.~\ref{sect:availability}, this +part of the specification was dropped in version 1.2. VO service standards previous to VOSI may not be forced to retrospectively implement VOSI (although that should be encouraged). @@ -114,14 +118,6 @@ \subsection{Role within the VO Architecture} \ref{appendix:harvesting} for a more detailed description of this use case.) -Once a user discovers data and services of interest, she will want to -engage them in an analysis process. Success requires that the selected -services are actually up and running properly as a down service can -cause automated processing to fail completely. Registry and workflow -services can assist with this by tracking the availability of services -and alerting users about downtime. We envision that VOSI will allow VO -projects to better track the overall health of the VO ecosystem. - \section{Interface bindings} \label{sec:bindings} @@ -138,9 +134,10 @@ \section{Interface bindings} protocol. The response to an HTTP POST, PUT or DELETE to these resources is not defined by this specification. However, if an implementation has no special action to perform for these requests, the normal response -would be a HTTP 405 "Method not allowed" status code. Both the -capabilities and availability bindings must be available to anonymous -requests. +would be a HTTP 405 "Method not allowed" status code. The +capabilities binding must be available to anonymous +requests.\todo{This is probably no longer true (see SSO) -- material for +another PR} The endpoints and interface types for the support interface shall be defined in the service's registration using one \xmlel{capability} @@ -151,7 +148,8 @@ \section{Interface bindings} When using the REST binding, any HTTP URLs may be used. The client must find the appropriate URLs from the service's entry in the VO registry and, in general, should not try and infer the URLs from any other URLs -for that service. However, standards for specific services may put extra +for that service.\todo{Which is in a bit of a conflict with the +introducton's ``by some other means''. Resolve this in another PR?} However, standards for specific services may put extra constraints on the form of the URLs. \section{Metadata specification} @@ -167,7 +165,10 @@ \section{Metadata specification} \item those describing other aspects of the service \end{itemize} -This section defines how each of these classes is represented. The +This section defines how each of these classes is represented.\todo{I'd +say let's switch this to our canonical prefixes as in other Registry +documents (vr:, vs:, \dots). It's more readable and less of a +typographic chore.} The following typographic convention is used to represent a XML element defined within a particular namespace: @@ -205,7 +206,7 @@ \subsection{Capability metadata} another the TAP implementation but these two might well apply to the same service. -An entry for a service in the resource registry - i.e., its VOResource - +An entry for a service in the resource registry -- i.e., its VOResource -- contains the Dublin Core resource metadata (identifier, curation information, content description, etc.) followed by the service's capability descriptions (expressed as a series of @@ -262,56 +263,22 @@ \subsection{Capability metadata} All VO services should provide this interface. \subsection{Availability metadata} +\label{sect:availability} -This interface indicates whether the service is operable and the -reliability of the service for extended and scheduled requests. The -availability shall be represented as an XML document in which the root -element is -\xmlel{http://www.ivoa.net/xml/Availability/v1.0\#availability}. This -element shall contain child elements providing the following -information: - -\begin{itemize} -\item \xmlel{available} -- whether the service is currently accepting requests -\item \xmlel{upSince} -- duration for which the service has been -continuously available -\item \xmlel{downAt} -- the instant at which the service is next -scheduled to be unavailable -\item \xmlel{backAt} -- the instant at which the service is scheduled to -become available again after down time; -\item \xmlel{note} -- textual note, e.g. explaining the reason for unavailability. -\end{itemize} - -The elements \xmlel{upSince}, \xmlel{downAt}, \xmlel{backAt} and -\xmlel{note} are optional. The \xmlel{available} element is mandatory. -There may be more than one \xmlel{note} element. - -The XML document shall conform to the schema given in appendix -\ref{appendix:availability} of this specification. - -When reporting availability, the service should do a good check on its -underlying parts to see if it is still operational and not just make a -simple return from a web server, e.g., if it relies on a database it -should check that the database is still up. If any of these checks fail, -the service should set available to false in the availability output. - -If a service is to be online but unavailable for work (e.g., when a -service with a work queue intends to shut down after draining the queue) -then the service should set available to false. - -There are no special elements in the availability document for the -contact details of the service operator. These details may be given as a -note element if they are known to the service. +Versions 1.0 and 1.1 of this specification required services to declare +information on their current and future availability. It was found that +this specification was insufficient to work with multi-capability and/or +mirrored services \citep{note:caproles}. Also, the interface has not +found the use that was originally anticipated. Therefore, availability +reporting is no longer part of VOSI, although the standards record still +defines a corresponding standards key, indicating support for the legacy +endpoint. -In the REST binding, the availability shall be a single web resource -with a registered URL - -All VO services shall provide this interface. \subsection{Table metadata} Some services deal with tabular data. These data may be the target of - ADQL queries, as in TAP \citep{2017ivoa.spec.0509L}, or they may be the + ADQL queries, as in TAP \citep{2019ivoa.spec.0927D}, or they may be the output of other operations, as in SIAP queries. In each case, it is useful if the service describes the details of the tables concerned. It is more useful if this description can be captured in the resource @@ -394,29 +361,22 @@ \subsection{Non-service metadata (non-normative)} \section{Registration of VOSI endpoints} \label{sec:endpoints} -The endpoints for the service and availability metadata shall be +The endpoints for the service metadata shall be included in the registration of each service that provides them. \begin{tabular}{l l l l l} \label{tab:registration} Endpoint type & standardID value \\ -availability & \nolinkurl{ivo://ivoa.net/std/VOSI#availability} \\ capabilities & \nolinkurl{ivo://ivoa.net/std/VOSI#capabilities} \\ tables (1.0) & \nolinkurl{ivo://ivoa.net/std/VOSI#tables} \\ -tables (1.1) & \nolinkurl{ivo://ivoa.net/std/VOSI#tables-1.1} \\ +tables (1.1 and 1.2) & \nolinkurl{ivo://ivoa.net/std/VOSI#tables-1.1} \\ \end{tabular} -An availability endpoint shall be represented by an element named -\xmlel{capability}, of type -\xmlel{http://www.ivoa.net/xml/VOResource/v1.0\#Capability} (defined by -the standard VOResource XML schema \citet{2018ivoa.spec.0625P}). The -value of the \xmlel{standardID} attribute of the capability shall be -\nolinkurl{ivo://ivoa.net/std/VOSI#availability}. - A capabilities endpoint should be represented by an element named \xmlel{capability}, of type \xmlel{http://www.ivoa.net/xml/VOResource/v1.0\#Capability}. If such a -capability is provided then the value of the \xmlel{standardID} +capability\todo{Since we require this endpoint, isn't this qualification +a bit silly?} is provided then the value of the \xmlel{standardID} attribute must be \nolinkurl{ivo://ivoa.net/std/VOSI#capabilities}. A tables endpoint should be represented by an element named @@ -426,7 +386,7 @@ \section{Registration of VOSI endpoints} attribute must be \nolinkurl{ivo://ivoa.net/std/VOSI\#tables}, or, for version 1.1, \nolinkurl{ivo://ivoa.net/std/VOSI\#tables-1.1}. -With all three VOSI functions, the \xmlel{capability} element that +With both VOSI functions, the \xmlel{capability} element that describes the function must contain an \xmlel{interface} element of a type semantically appropriate for the binding of the function to the service; the \xmlel{accessURL} element within the \xmlel{interface} @@ -438,9 +398,9 @@ \section{Registration of VOSI endpoints} type to encode VOSI endpoints (see the examples given in section~\ref{sec:examples}). -The capabilities and availability endpoints must not require any -credentials to view. Thus, the \xmlel{interface} registry entries for -capabilities and availability must not contain a securityMethod element. +The capabilities endpoint must not require any +credentials to view\todo{See above}. Thus, the \xmlel{interface} registry entries for +capabilities must not contain a securityMethod element. \section{Example VOSI responses} \label{sec:examples} @@ -515,15 +475,6 @@ \subsection{Example 1: SIA 1.0 capabilities} - - - - - http://adil.ncsa.uiuc.edu/cgi-bin/voimquery/availability - - - - \end{lstlisting} @@ -688,163 +639,6 @@ \subsection{Example 4: Child table resource} \end{lstlisting} -\subsection{Example 5: Availability} - -A sample response from an availability resource. - -\begin{lstlisting}[language=XML] - - true - service is accepting queries - -\end{lstlisting} - -\appendix - -\section{The Complete VOSICapabilities Schema} -\label{appendix:capabilities} -\begin{lstlisting}[language=XML] - - - - - A schema for formatting service capabilities as returned by a - capabilities resource, defined by the IVOA Support Interfaces - specification (VOSI). - - See http://www.ivoa.net/Documents/latest/VOSI.html. - - - - - - - - - - - A listing of capabilities supported by a service - - - - - - - - - - A capability supported by the service. - - - - A protocol-specific capability is included by specifying a - vr:Capability sub-type via an xsi:type attribute on this - element. - - - - - - - - - - - - -\end{lstlisting} - -\section{The Complete VOSIAvailability Schema} -\label{appendix:availability} -\begin{lstlisting}[language=XML] - - - - - A schema for formatting availability metadata as returned by an - availability resource defined in the IVOA Support Interfaces - specification (VOSI). - - See http://www.ivoa.net/Documents/latest/VOSI.html. - - - - - - - - - - - - - Indicates whether the service is currently available. - - - - - - - - The instant at which the service last became available. - - - - - - - - The instant at which the service is next scheduled to become - unavailable. - - - - - - - - The instant at which the service is scheduled to become available - again after a period of unavailability. - - - - - - - - A textual note concerning availability. - - - - - - - - - -\end{lstlisting} - \section{The Complete VOSITables Schema} \label{appendix:tables} \begin{lstlisting}[language=XML] @@ -1005,6 +799,13 @@ \section{Use Case for Capability Harvesting (non-normative)} \section{Changes from Previous Versions} \label{appendix:changes} +\subsection{Changes since REC-1.1} + +\begin{itemize} +\item Removed specification for the availability endpoint; added a brief +discussion of the rationale. +\end{itemize} + \subsection{Changes since REC-VOSI-20110531} Added alternate root element (table) to VOSITables schema. diff --git a/local.bib b/local.bib new file mode 100644 index 0000000..127bacf --- /dev/null +++ b/local.bib @@ -0,0 +1,8 @@ +@MISC{note:caproles, + author={Demleitner, M. and Harrison, P.}, + title={On the Use of Capabilities in the {VO}}, + year=2019, + month=Mar, + howpublished={{IVOA Note, 15 March 2019}}, + url={http://ivoa.net/documents/caproles/} +} From 45499e6300b1404463e473e39860a4c9ead5329a Mon Sep 17 00:00:00 2001 From: Markus Demleitner Date: Wed, 16 Feb 2022 09:39:17 +0100 Subject: [PATCH 03/15] Adding a StandardsRegExt record for VOSI. Well, it turns out there hasn't been one all these years, which means that all VOSI StandardIDs ("ivo://ivoa.net/std/VOSI#capabilities") really pointed nowhere. That nobody even noticed may raise questions on how useful StandardsRegExt StandardKeys actually are. But that's beside the point here. I need the record here to say something about what #availability is in a post-caproles VO. --- VOSI.vor | 100 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 VOSI.vor diff --git a/VOSI.vor b/VOSI.vor new file mode 100644 index 0000000..f4eb6a4 --- /dev/null +++ b/VOSI.vor @@ -0,0 +1,100 @@ + + + IVOA Support Interfaces + VOSI + ivo://ivoa.net/std/VOSI + doi:10.5479/ADS/bib/2017ivoa.spec.0524G + + IVOA + + + Graham, Matthew + + + Rixon, Guy + + + Dowler, Patrick + + + Major, Brian + + + 2017-05-24 + 1.1 + + IVOA GWS WG + grid@ivoa.net + + + + virtual-observatories + + + This document describes the minimum interface that a web service requires + to participate in the IVOA. Note that this is not required of standard VO + services developed prior to this specification, although uptake is + strongly encouraged on any subsequent revision. All new standard VO + services, however, must feature a VOSI-compliant interface. + + https://ivoa.net/documents/VOSI/ + Other + Research + + + 1.1 + + + http://www.ivoa.net/xml/VOSICapabilities/v1.0 + + + http://www.ivoa.net/xml/VOSIAvailability/v1.0 + This schema is no longer part of the normative content. + It is retained for backward compatiblilty, though. + + + http://www.ivoa.net/xml/VOSITables/v1.0 + + + + availability + Legacy standardID for capabilities describing the + service availability endpoint defined in VOSI 1.0 and VOSI 1.1. + + + + capabilities + standardID for capabilities describing the + endpoint for retrieving VOSI capability metadata about a service. + + + + tables + standardID for capabilities describing + endpoints for retrieving VOSI tables metadata compliant to VOSI + version 1.0. + + + + tables-1.0 + standardID for capabilities describing + endpoints for retrieving VOSI tables metadata compliant to VOSI + version 1.1 and later; over #tables, these react to the details + parameter. + + + From ba0baf12bd78195d8c23d4e87a10eb88877d6443 Mon Sep 17 00:00:00 2001 From: Markus Demleitner Date: Wed, 23 Feb 2022 16:06:10 +0100 Subject: [PATCH 04/15] Restoring VOSICapabilities in the printed document. Also changing the inclusion of the VOSITables schema to lstinputlisting; pulling this from the authoritative, on-disk version seems a good idea. Whether it's a good idea to print the two schema files at all is another matter. I'd say no, but I'd leave removing them to the editor. --- VOSI.tex | 57 ++++++++------------------------------------------------ ivoatex | 2 +- 2 files changed, 9 insertions(+), 50 deletions(-) diff --git a/VOSI.tex b/VOSI.tex index e0ef985..4152ebd 100644 --- a/VOSI.tex +++ b/VOSI.tex @@ -639,58 +639,17 @@ \subsection{Example 4: Child table resource} \end{lstlisting} + +\appendix +\section{The Complete VOSICapabilities Schema} +\label{appendix:capabilities} + +\lstinputlisting[language=XML]{VOSICapabilities-v1.0.xsd} + \section{The Complete VOSITables Schema} \label{appendix:tables} -\begin{lstlisting}[language=XML] - - - < xsd:documentation> - A schema for formatting table metadata as returned by a - tables resource, defined by the IVOA Support Interfaces - specification (VOSI). - - See http://www.ivoa.net/Documents/latest/VOSI.html. - - - - - - - - - - A description of the table metadata supported by the - service associated with a VOSI-enabled resource. - - - - - - - - - A description of a single table supported by the - service associated with a VOSI-enabled resource. - - - - - -\end{lstlisting} +\lstinputlisting[language=XML]{VOSITables-v1.1.xsd} \section{Use Case for Capability Harvesting (non-normative)} \label{appendix:harvesting} diff --git a/ivoatex b/ivoatex index 33e1ef3..4349111 160000 --- a/ivoatex +++ b/ivoatex @@ -1 +1 @@ -Subproject commit 33e1ef3343f50a1ff56061df5ccdc5b59819cbce +Subproject commit 4349111224c732b1ece6e77073e55419199732ac From 41aed7f640bffb02b9bd672158fb4a286a047eef Mon Sep 17 00:00:00 2001 From: Markus Demleitner Date: Wed, 23 Feb 2022 16:10:21 +0100 Subject: [PATCH 05/15] Updating Makefile to auto-pull ivoatex if necessary. Also including git metadata. --- Makefile | 9 +++++++-- VOSI.tex | 2 ++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 913cdff..e90629b 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ AUTHOR_EMAIL = msdemlei@ari.uni-heidelberg.de # Source files for the TeX document (but the main file must always # be called $(DOCNAME).tex -SOURCES = $(DOCNAME).tex role_diagram.pdf +SOURCES = $(DOCNAME).tex role_diagram.pdf gitmeta.tex # List of pixel image files to be included in submitted package FIGURES = role_diagram.svg @@ -27,4 +27,9 @@ VECTORFIGURES = # Additional files to distribute (e.g., CSS, schema files, examples...) AUX_FILES = -include ivoatex/Makefile +-include ivoatex/Makefile + +ivoatex/Makefile: + @echo "*** ivoatex submodule not found. Initialising submodules." + @echo + git submodule update --init diff --git a/VOSI.tex b/VOSI.tex index 4152ebd..c7256ca 100644 --- a/VOSI.tex +++ b/VOSI.tex @@ -1,6 +1,8 @@ \documentclass[11pt,letter]{ivoa} \input tthdefs +\input gitmeta + \usepackage{listings} \lstloadlanguages{XML} \lstset{flexiblecolumns=true,basicstyle=\footnotesize,tagstyle=\ttfamily,showspaces=false} From 09d495c1d44ddedda31399c25acb3f37937444ba Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Sat, 9 Nov 2024 15:57:55 -0800 Subject: [PATCH 06/15] switch to WD-VOSI-1.2 initial user-managed table extension in OpenAPI --- Makefile | 7 ++-- VOSI.tex | 8 ++-- openapi/vosi-table.yaml | 83 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index 913cdff..3dce662 100644 --- a/Makefile +++ b/Makefile @@ -4,15 +4,16 @@ DOCNAME = VOSI # count up; you probably do not want to bother with versions <1.0 -DOCVERSION = 1.1 +DOCVERSION = 1.2 # Publication date, ISO format; update manually for "releases" -DOCDATE = 2022-03-10 +DOCDATE = 2024-11-11 # What is it you're writing: NOTE, WD, PR, or REC DOCTYPE = WD -AUTHOR_EMAIL = msdemlei@ari.uni-heidelberg.de +# for push to doc repo +AUTHOR_EMAIL = pdowler.cadc@gmail.com # Source files for the TeX document (but the main file must always # be called $(DOCNAME).tex diff --git a/VOSI.tex b/VOSI.tex index 944c5f3..dee2137 100644 --- a/VOSI.tex +++ b/VOSI.tex @@ -9,12 +9,12 @@ \ivoagroup{Grid and Web Services Working Group} -\author{Grid and Web Services Working Group} +\author{Patrick Dowler} +\author{Matthew Graham} +\author{Brian Major} +\author{Guy Rixon} -\editor{Matthew Graham} -\editor{Guy Rixon} \editor{Patrick Dowler} -\editor{Brian Major} \previousversion[https://ivoa.net/documents/VOSI/20170524/]{REC-1.1} \previousversion[http://www.ivoa.net/documents/VOSI/20110531]{REC-1.0} diff --git a/openapi/vosi-table.yaml b/openapi/vosi-table.yaml index fda1c9e..93fa105 100644 --- a/openapi/vosi-table.yaml +++ b/openapi/vosi-table.yaml @@ -18,6 +18,78 @@ get: $ref: ./vosi-std-responses.yaml#/permission-denied 404: $ref: ./vosi-std-responses.yaml#/not-found +post: + summary: proto - update table metadata + description: | + update the tap_schema metadata from description; + TBD: what can be changed and what is immutable? + parameters: + - name: name + in: path + description: the fully qualified name of the table to create + required: true + schema: + type: string + requestBody: + $ref: '#/components/schemas/tableDoc' + responses: + 204: + description: table updated + 400: + description: update table metadata failed due to invalid input + content: + text/plain: + 401: + $ref: ./vosi-std-responses.yaml#/not-authenticated + 403: + $ref: ./vosi-std-responses.yaml#/permission-denied + 404: + $ref: ./vosi-std-responses.yaml#/not-found +put: + summary: proto - create table + description: create table from description + parameters: + - name: name + in: path + description: the fully qualified name of the table to create + required: true + schema: + type: string + requestBody: + $ref: '#/components/schemas/tableDoc' + responses: + 201: + description: table created + 400: + description: create table failed due to invalid input + content: + text/plain: + 401: + $ref: ./vosi-std-responses.yaml#/not-authenticated + 403: + $ref: ./vosi-std-responses.yaml#/permission-denied + 404: + $ref: ./vosi-std-responses.yaml#/not-found +delete: + summary: proto - delete table + parameters: + - name: name + in: path + description: | + the table to delete; + TBD: remove from tap_schema and drop from db? + required: true + schema: + type: string + responses: + 204: + description: table dropped + 401: + $ref: ./vosi-std-responses.yaml#/not-authenticated + 403: + $ref: ./vosi-std-responses.yaml#/permission-denied + 404: + $ref: ./vosi-std-responses.yaml#/not-found components: schemas: @@ -30,3 +102,14 @@ components: name: table prefix: vosi namespace: http://www.ivoa.net/xml/VOSITables/v1.0 + ## Note: adding the VOTable type below has these effects: + # put: allows VOTable to be used to create a table, empty only? with rows? + # post: allows VOTable to be used to update table metadata, empty only? add rows? + # get: allows caller to request table description as an empty VOTable + # there are some subtle differences in metadata content + application/x-votable+xml: + schema: + type: object + xml: + name: VOTABLE + namespace: http://www.ivoa.net/xml/VOTable/v1.3 From 8c7278d70a321fd87f98ae59c92398a34af36cab Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Fri, 29 Nov 2024 13:21:33 -0800 Subject: [PATCH 07/15] specify optional vosi-table actions as 405 response code --- openapi/vosi-table.yaml | 40 +++++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 13 deletions(-) diff --git a/openapi/vosi-table.yaml b/openapi/vosi-table.yaml index 93fa105..366e71b 100644 --- a/openapi/vosi-table.yaml +++ b/openapi/vosi-table.yaml @@ -10,7 +10,7 @@ get: type: string responses: 200: - description: metadata for the specified table + description: metadata for the specified schema or table $ref: '#/components/schemas/tableDoc' 401: $ref: ./vosi-std-responses.yaml#/not-authenticated @@ -19,10 +19,8 @@ get: 404: $ref: ./vosi-std-responses.yaml#/not-found post: - summary: proto - update table metadata - description: | - update the tap_schema metadata from description; - TBD: what can be changed and what is immutable? + summary: proto - update table metadata and/or content (optional) + description: update the tap_schema metadata from a table description; may include rows to append if input format supports it parameters: - name: name in: path @@ -39,15 +37,23 @@ post: description: update table metadata failed due to invalid input content: text/plain: + schema: + type: string 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: $ref: ./vosi-std-responses.yaml#/permission-denied 404: $ref: ./vosi-std-responses.yaml#/not-found + 405: + description: this response indicates that the service does not support update of tables + content: + text/plain: + schema: + type: string put: - summary: proto - create table - description: create table from description + summary: proto - create table (optional) + description: create table from description; may include rows if input format supports it parameters: - name: name in: path @@ -70,8 +76,15 @@ put: $ref: ./vosi-std-responses.yaml#/permission-denied 404: $ref: ./vosi-std-responses.yaml#/not-found + 405: + description: this response indicates that the service does not support creation of tables by users + content: + text/plain: + schema: + type: string delete: - summary: proto - delete table + summary: proto - delete table (optional) + description: delete table by name parameters: - name: name in: path @@ -90,6 +103,12 @@ delete: $ref: ./vosi-std-responses.yaml#/permission-denied 404: $ref: ./vosi-std-responses.yaml#/not-found + 405: + description: this response indicates that the service does not support deletion of tables created by users + content: + text/plain: + schema: + type: string components: schemas: @@ -102,11 +121,6 @@ components: name: table prefix: vosi namespace: http://www.ivoa.net/xml/VOSITables/v1.0 - ## Note: adding the VOTable type below has these effects: - # put: allows VOTable to be used to create a table, empty only? with rows? - # post: allows VOTable to be used to update table metadata, empty only? add rows? - # get: allows caller to request table description as an empty VOTable - # there are some subtle differences in metadata content application/x-votable+xml: schema: type: object From e5d2ee9d3cd8c2b0b9a94fc2d4a057d84d0c81e0 Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Thu, 8 May 2025 14:05:39 -0700 Subject: [PATCH 08/15] Add/Update gh-Workflows for PDF Preview --- .github/workflows/build.yml | 45 ++++++++++++++++++++++++ .github/workflows/preview.yml | 66 +++++++++++++++++++++++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 .github/workflows/build.yml create mode 100644 .github/workflows/preview.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml new file mode 100644 index 0000000..7f015bf --- /dev/null +++ b/.github/workflows/build.yml @@ -0,0 +1,45 @@ +# This file generated from a template file maintained in the ivoatex repository. +# To create and install it into a project repository, do: +# make github-preview +# git commit +# git push +# +name: Check the IVOA document + +env: + doc_name: VOSI + +on: + pull_request: + types: [opened, synchronize, reopened] + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + + - name: Checkout the repository + uses: actions/checkout@v1 + with: + submodules: true + + - name: Setup dependencies + run: | + sudo apt update + sudo apt install texlive-latex-base texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended xsltproc latexmk cm-super + + - name: Build the document + run: make + + - name: Check the output + run: | + test -f ${{ env.doc_name }}.pdf + test -f ${{ env.doc_name }}.bbl + + - name: Keep the PDF artefact + uses: actions/upload-artifact@v1 + with: + name: PDF Preview + path: ${{ env.doc_name }}.pdf diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml new file mode 100644 index 0000000..a410263 --- /dev/null +++ b/.github/workflows/preview.yml @@ -0,0 +1,66 @@ +# This file generated from a template file maintained in the ivoatex repository. +# To create and install it into a project repository, do: +# make github-preview +# git commit +# git push +# +name: Update PDF Preview + +env: + doc_name: VOSI + +on: + push: + branches: + - master + +jobs: + build: + + runs-on: ubuntu-latest + + steps: + + - name: Checkout the repository + uses: actions/checkout@v1 + with: + submodules: true + + - name: Setup dependencies + run: | + sudo apt update + sudo apt install texlive-latex-base texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended xsltproc latexmk cm-super + sudo snap install pdftk + + - name: Build the document + run: make ${{ env.doc_name }}-draft.pdf + + - name: Check the output + run: | + test -f ${{ env.doc_name }}-draft.pdf + test -f ${{ env.doc_name }}.bbl + + - name: Move the auto-pdf-preview tag + uses: weareyipyip/walking-tag-action@v2 + with: + tag-name: auto-pdf-preview + tag-message: | + Last commit taken into account for the automatically updated PDF preview of this IVOA document. + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Update the PDF preview + uses: Xotl/cool-github-releases@v1 + with: + mode: update + isPrerelease: true + tag_name: auto-pdf-preview + release_name: "Auto PDF Preview" + body_mrkdwn: | + This release aims to provide a PDF preview of the last commit applied on this repository. + It will be updated automatically after each merge of a PullRequest. + **DO NOT PUBLISH THIS PRE-RELEASE!**" + _Corresponding commit: ${{ github.sha }}_ + assets: ${{ env.doc_name }}-draft.pdf + replace_assets: true + github_token: ${{ secrets.GITHUB_TOKEN }} From 987086b1475fd1b2e8986d0d0bce4b1ed808d289 Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Thu, 8 May 2025 14:16:55 -0700 Subject: [PATCH 09/15] remove openapi description of availability --- openapi/vosi-availability.yaml | 36 ---------------------------------- 1 file changed, 36 deletions(-) delete mode 100644 openapi/vosi-availability.yaml diff --git a/openapi/vosi-availability.yaml b/openapi/vosi-availability.yaml deleted file mode 100644 index e07345c..0000000 --- a/openapi/vosi-availability.yaml +++ /dev/null @@ -1,36 +0,0 @@ -head: - tag: VOSI - summary: VOSI Availability heartbeat check - description: | - Perform a minimal check that the service is reachable and running. This - check is intended to be very fast and lightweight. - responses: - 200: - description: service is reachable -get: - tag: VOSI - summary: VOSI Availability full check - description: | - Perform a complete check that the service is operational. This normally - includes checking the availability of back end resources (e.g. database, - storage, ...) and that other services that this service depends on (e.g. - AAI services) are reachable (via heartbeat availability check). - responses: - 200: - description: A VOSI availability document in XML. - content: - text/xml: - schema: - $ref: '#/components/schemas/availability' - -components: - schemas: - availability: - content: - text/xml: - schema: - type: object - xml: - name: availability - prefix: vosi - namespace: http://www.ivoa.net/xml/VOSIAvailability/v1.0 From 08ea8a73823b8987ecbd3eb8eb8558594dec42eb Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Thu, 8 May 2025 17:38:17 -0700 Subject: [PATCH 10/15] update tables section of document to refer to the OpenAPI spec trying to make the prose less specification and more explanation and examples --- VOSI.tex | 107 +++++++++++++++++++++++++------- openapi/vosi-capabilities.yaml | 5 +- openapi/vosi-std-responses.yaml | 33 +++++++--- openapi/vosi-table.yaml | 63 ++++++++++--------- openapi/vosi-tableset.yaml | 3 +- 5 files changed, 149 insertions(+), 62 deletions(-) diff --git a/VOSI.tex b/VOSI.tex index d077ec8..526c7a4 100644 --- a/VOSI.tex +++ b/VOSI.tex @@ -279,12 +279,12 @@ \subsection{Availability metadata} \subsection{Table metadata} - Some services deal with tabular data. These data may be the target of - ADQL queries, as in TAP \citep{2019ivoa.spec.0927D}, or they may be the - output of other operations, as in SIAP queries. In each case, it is - useful if the service describes the details of the tables concerned. It - is more useful if this description can be captured in the resource - registry. +Some services deal with tabular data. These data may be the target of +ADQL queries, as in TAP \citep{2019ivoa.spec.0927D}, or they may be the +output of other operations, as in SIAP queries. In each case, it is +useful if the service describes the details of the tables concerned. It +is more useful if this description can be captured in the resource +registry. The VODataService standard \citep{2021ivoa.spec.1102D} defines XML elements for describing a set of tables. These elements can be included @@ -299,28 +299,38 @@ \subsection{Table metadata} XML document of which the root element is of type\\ $$\hbox{\xmlel{http://www.ivoa.net/xml/VODataService/v1.1\#Table}}$$\\ This element may contain any mix of elements allowed by the VODataService XML schema. +The table metadata for a single table can also be representated using an empty VOTable. -In the REST binding, the TableSet metadata shall be a hierarchical web -resource with a registered URL. There are two levels of TableSet -metadata detail that may be returned from the base \textit{tables} -endpoint. Maximum detail (max) is the complete metadata including all -details of the Table elements. Minimum detail (min) includes the full -set of Table elements with names and optional descriptions, but does not -include Table elements of type Column or ForeignKey. +The REST interface for the tables endpoint in described in detail in the OpenAPI +specification (see ???). Here we give a brief summary of the main use and some +examples but defer to the OpenAPI documents as the definitive specification. -If the parameter \textit{detail} accompanies the request with values of -either \textit{max} or \textit{min}, the service should take this as a -suggestion as to what level of detail to return, but may return either -level. If the parameter \textit{detail} is not present, the service may -choose the level of metadata detail to return. +\subsubsection{Tables} +The TableSet metadata is a hierarchical web resource with two levels of detail. +Maximum detail (max) is the complete metadata including all details of the table +structure (columns and foreign keys). Minimum detail (min) includes the full +set of tables (organised by schema) with names and optional descriptions, but does not +include column of foreign key information. For example, the request: \begin{verbatim} -GET http://example.net/srv/tables?detail=min +GET http://example.net/srv/tables \end{verbatim} +\noindent would return the default (service-chosen) amount of detail (either just table +listing or the complete TableSet metadata). -should be taken as a suggestion to return the minimum level of TableSet +The caller can use the \verb|detail| parameter to request a specific amount of detail. +For example: +\begin{verbatim} +GET http://example.net/srv/tables?detail=min +\end{verbatim} +\noindent should be taken as a suggestion to return the minimum level of TableSet +metadata detail, while: +\begin{verbatim} +GET http://example.net/srv/tables?detail=max +\end{verbatim} +\noindent should be taken as a suggestion to return the maximum level of TableSet metadata detail. Services with a large number of tables and/or columns cannot normally @@ -342,9 +352,64 @@ \subsection{Table metadata} \begin{verbatim} ivoa.ObsCore + ... ... -\end{verbatim} +\end{verbatim} + +The above \xmlel{vosi:table} document is the default response format; clients +may request the alternate VOTable format using the HTTP \verb|accept| header with a +value of the VOTable mime-type (\verb|"application/x-votable+xml"|). + +\subsubsection{User-managed Tables} +Some services may allow users to create tables using the tables endpoint. The lifetime +of such tables and the authentication and authorization requirements for using this feature +are an implementation detail that may vary from one service to another. At one extreme, +users may be able to create tables anonymously and the table will persist for a short +perioid of time (days or weeks) before being removed. At the other extreme, users would +have to authenticate, have permission to create tables in a specific schema, and the tables +would persist forever. + +The visibility of user-managed tables is also an implementation detail not specified here. +Expectation: short-lived anonymously created tables would not be visible (e.g. not listed +by the tables endpoint of the service or be added to the \verb|tap_schema| of a TAP service). +Expectation: permanent tables created (owned) by an authenticated user could be made +visible to and usable by other users (e.g. included in tables endpoint output and added to +the \verb|tap_schema| of a TAP service). + +To create a new table: +\begin{verbatim} +PUT http://example.net/srv/tables/schema1.table1 +\end{verbatim} +\noindent where the \verb|| payload can be either a VOSI-table or VOTable document. + +To update table metadata: +\begin{verbatim} +POST http://example.net/srv/tables/schema1.table1 +\end{verbatim} +\noindent where only some of the table metadata can me modified (see below). In addition, if +the VOTable format is used for updating metadata, the VOTable must not contain any rows. + +To delete a table: +\begin{verbatim} +DELETE http://example.net/srv/tables/schema1.table1 +\end{verbatim} + +\subsubsection{Mutable Table Metadata} +Metadata that can be updated: +\begin{itemize} +\item table metadata that can be updated: utype, description +\item column metadata that can be updated: utype, ucd, unit, description +\end{itemize} + +Column metadata where update would imply an structural change to the table (e.g. an SQL +alter table statement): +\begin{itemize} +\item column datatype, arraysize, xtype: alter column datatype? +\item input table description has fewer columns: drop column? +\item input table description has more columns: add column? +\end{itemize} +\noindent These are probably too fraught with pain in the details. \subsection{Non-service metadata (non-normative)} diff --git a/openapi/vosi-capabilities.yaml b/openapi/vosi-capabilities.yaml index 8177be5..3c6c395 100644 --- a/openapi/vosi-capabilities.yaml +++ b/openapi/vosi-capabilities.yaml @@ -1,4 +1,5 @@ head: + operationId: probe-capabilities tag: VOSI summary: VOSI Capabilities with SSO-next prototype description: | @@ -9,12 +10,13 @@ head: for available authentication methods. responses: 200: - description: successful probe, service allows request from caller + $ref: ./vosi-std-responses.yaml#/authenticated 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: $ref: ./vosi-std-responses.yaml#/permission-denied get: + operationId: read-capabilities tag: VOSI summary: VOSI Capabilities description: Get the set of VOResource capability descriptions for this service. @@ -25,6 +27,7 @@ get: text/xml: schema: $ref: '#/components/schemas/capabilities' + $ref: ./vosi-std-responses.yaml#/not-authenticated 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: diff --git a/openapi/vosi-std-responses.yaml b/openapi/vosi-std-responses.yaml index 636568e..305ecb5 100644 --- a/openapi/vosi-std-responses.yaml +++ b/openapi/vosi-std-responses.yaml @@ -1,19 +1,34 @@ # standard +authenticated: + description: a response header indicating that authentication was successful + headers: + x-vo-authenticated: + schema: + type: string + not-authenticated: description: authentication is required and current attempt failed headers: - www-authenticate: - schema: - type: string + www-authenticate: + schema: + type: string + permission-denied: description: caller not authorized to perform operation content: - text/plain: - schema: - type: string + text/plain: + schema: + type: string not-found: description: resource not found content: - text/plain: - schema: - type: string + text/plain: + schema: + type: string + +not-implemented: + description: this response indicates that the service does not support this operation + content: + text/plain: + schema: + type: string diff --git a/openapi/vosi-table.yaml b/openapi/vosi-table.yaml index 366e71b..de7afb4 100644 --- a/openapi/vosi-table.yaml +++ b/openapi/vosi-table.yaml @@ -1,16 +1,21 @@ get: + operationId: read-table-metadata + tag: VOSI-1.2 summary: get metadata for the specified table description: provides the table metadata for a single schema or table parameters: - - name: name - in: path - description: a single schema name or table name as provided in tap_schema - required: true - schema: - type: string + - name: name + in: path + description: a single schema name or table name as provided in tap_schema + required: true + schema: + type: string responses: 200: description: metadata for the specified schema or table + $ref: ./vosi-anon-read.yaml + $ref: ./vosi-group-read.yaml + $ref: ./vosi-group-write.yaml $ref: '#/components/schemas/tableDoc' 401: $ref: ./vosi-std-responses.yaml#/not-authenticated @@ -19,6 +24,8 @@ get: 404: $ref: ./vosi-std-responses.yaml#/not-found post: + operationId: update-table-metadata + tag: VOSI-1.2 summary: proto - update table metadata and/or content (optional) description: update the tap_schema metadata from a table description; may include rows to append if input format supports it parameters: @@ -28,6 +35,9 @@ post: required: true schema: type: string + - $ref: ./vosi-anon-read.yaml + - $ref: ./vosi-group-read.yaml + - $ref: ./vosi-group-write.yaml requestBody: $ref: '#/components/schemas/tableDoc' responses: @@ -46,21 +56,22 @@ post: 404: $ref: ./vosi-std-responses.yaml#/not-found 405: - description: this response indicates that the service does not support update of tables - content: - text/plain: - schema: - type: string + $ref: ./vosi-std-responses.yaml#/not-implemented put: + operationId: create-table + tag: VOSI-1.2 summary: proto - create table (optional) description: create table from description; may include rows if input format supports it parameters: - - name: name - in: path - description: the fully qualified name of the table to create - required: true - schema: - type: string + - name: name + in: path + description: the fully qualified name of the table to create + required: true + schema: + type: string + - $ref: ./vosi-anon-read.yaml + - $ref: ./vosi-group-read.yaml + - $ref: ./vosi-group-write.yaml requestBody: $ref: '#/components/schemas/tableDoc' responses: @@ -77,20 +88,16 @@ put: 404: $ref: ./vosi-std-responses.yaml#/not-found 405: - description: this response indicates that the service does not support creation of tables by users - content: - text/plain: - schema: - type: string + $ref: ./vosi-std-responses.yaml#/not-implemented delete: + operationId: delete-table + tag: VOSI-1.2 summary: proto - delete table (optional) description: delete table by name parameters: - name: name in: path - description: | - the table to delete; - TBD: remove from tap_schema and drop from db? + description: the table to delete required: true schema: type: string @@ -104,11 +111,7 @@ delete: 404: $ref: ./vosi-std-responses.yaml#/not-found 405: - description: this response indicates that the service does not support deletion of tables created by users - content: - text/plain: - schema: - type: string + $ref: ./vosi-std-responses.yaml#/not-implemented components: schemas: diff --git a/openapi/vosi-tableset.yaml b/openapi/vosi-tableset.yaml index 4768a69..f1873ee 100644 --- a/openapi/vosi-tableset.yaml +++ b/openapi/vosi-tableset.yaml @@ -1,10 +1,11 @@ get: + operationId: get-tableset tag: VOSI tag: table metadata summary: VOSI Tables description: Provides the table metadata in the form of a TableSet description parameters: - - name: DETAIL + - name: detail in: query description: level of detail requested required: false From 2913f8bcf89ffad4bf867bb4459222fffd159def Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Fri, 9 May 2025 13:15:36 -0700 Subject: [PATCH 11/15] refine openapi spec so it can be pulled into another openapi spec --- ...enapi-example.yaml => openapi-example.yaml | 0 openapi/vosi-capabilities.yaml | 19 +++--- openapi/vosi-std-responses.yaml | 8 +++ openapi/vosi-table.yaml | 58 ++++++++----------- openapi/vosi-tableset.yaml | 8 +-- 5 files changed, 47 insertions(+), 46 deletions(-) rename openapi/openapi-example.yaml => openapi-example.yaml (100%) diff --git a/openapi/openapi-example.yaml b/openapi-example.yaml similarity index 100% rename from openapi/openapi-example.yaml rename to openapi-example.yaml diff --git a/openapi/vosi-capabilities.yaml b/openapi/vosi-capabilities.yaml index 3c6c395..145da31 100644 --- a/openapi/vosi-capabilities.yaml +++ b/openapi/vosi-capabilities.yaml @@ -1,6 +1,7 @@ head: operationId: probe-capabilities - tag: VOSI + tags: + - VOSI Capabilities summary: VOSI Capabilities with SSO-next prototype description: | Get standard headers for the service. The standard headers should include @@ -14,28 +15,28 @@ head: 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: + $ref: ./vosi-std-responses.yaml#/authenticated $ref: ./vosi-std-responses.yaml#/permission-denied get: operationId: read-capabilities - tag: VOSI - summary: VOSI Capabilities + tags: + - VOSI + summary: VOSI Capabilities with SSO-next prototype description: Get the set of VOResource capability descriptions for this service. responses: 200: - description: A VOSI Capabilities document in XML. - content: - text/xml: - schema: - $ref: '#/components/schemas/capabilities' - $ref: ./vosi-std-responses.yaml#/not-authenticated + $ref: '#/components/schemas/capabilities' + $ref: ./vosi-std-responses.yaml#/authenticated 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: + $ref: ./vosi-std-responses.yaml#/authenticated $ref: ./vosi-std-responses.yaml#/permission-denied components: schemas: capabilities: + description: A VOSI Capabilities document content: text/xml: schema: diff --git a/openapi/vosi-std-responses.yaml b/openapi/vosi-std-responses.yaml index 305ecb5..040e498 100644 --- a/openapi/vosi-std-responses.yaml +++ b/openapi/vosi-std-responses.yaml @@ -19,6 +19,14 @@ permission-denied: text/plain: schema: type: string + +bad-request: + description: invalid request/input + content: + text/plain: + schema: + type: string + not-found: description: resource not found content: diff --git a/openapi/vosi-table.yaml b/openapi/vosi-table.yaml index de7afb4..0589c17 100644 --- a/openapi/vosi-table.yaml +++ b/openapi/vosi-table.yaml @@ -1,21 +1,18 @@ get: operationId: read-table-metadata - tag: VOSI-1.2 + tags: + - VOSI table metadata summary: get metadata for the specified table description: provides the table metadata for a single schema or table parameters: - - name: name - in: path + - in: path + name: name description: a single schema name or table name as provided in tap_schema required: true schema: type: string responses: 200: - description: metadata for the specified schema or table - $ref: ./vosi-anon-read.yaml - $ref: ./vosi-group-read.yaml - $ref: ./vosi-group-write.yaml $ref: '#/components/schemas/tableDoc' 401: $ref: ./vosi-std-responses.yaml#/not-authenticated @@ -25,30 +22,25 @@ get: $ref: ./vosi-std-responses.yaml#/not-found post: operationId: update-table-metadata - tag: VOSI-1.2 + tags: + - VOSI table metadata summary: proto - update table metadata and/or content (optional) - description: update the tap_schema metadata from a table description; may include rows to append if input format supports it + description: update the table metadata from a table description; may include rows to append if input format supports it parameters: - - name: name - in: path + - in: path + name: name description: the fully qualified name of the table to create required: true schema: type: string - - $ref: ./vosi-anon-read.yaml - - $ref: ./vosi-group-read.yaml - - $ref: ./vosi-group-write.yaml requestBody: + description: a table description $ref: '#/components/schemas/tableDoc' responses: 204: description: table updated 400: - description: update table metadata failed due to invalid input - content: - text/plain: - schema: - type: string + $ref: ./vosi-std-responses.yaml#/bad-request 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: @@ -59,28 +51,25 @@ post: $ref: ./vosi-std-responses.yaml#/not-implemented put: operationId: create-table - tag: VOSI-1.2 + tags: + - VOSI table metadata summary: proto - create table (optional) - description: create table from description; may include rows if input format supports it + description: create table from a table description; may include rows if input format supports it parameters: - - name: name - in: path + - in: path + name: name description: the fully qualified name of the table to create required: true schema: - type: string - - $ref: ./vosi-anon-read.yaml - - $ref: ./vosi-group-read.yaml - - $ref: ./vosi-group-write.yaml + type: string requestBody: + description: a table description $ref: '#/components/schemas/tableDoc' responses: 201: - description: table created + description: table created 400: - description: create table failed due to invalid input - content: - text/plain: + $ref: ./vosi-std-responses.yaml#/bad-request 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: @@ -91,7 +80,8 @@ put: $ref: ./vosi-std-responses.yaml#/not-implemented delete: operationId: delete-table - tag: VOSI-1.2 + tags: + - VOSI table metadata summary: proto - delete table (optional) description: delete table by name parameters: @@ -103,7 +93,7 @@ delete: type: string responses: 204: - description: table dropped + description: table dropped 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: @@ -116,6 +106,7 @@ delete: components: schemas: tableDoc: + description: metadata for the specified schema or table content: text/xml: schema: @@ -130,3 +121,4 @@ components: xml: name: VOTABLE namespace: http://www.ivoa.net/xml/VOTable/v1.3 + diff --git a/openapi/vosi-tableset.yaml b/openapi/vosi-tableset.yaml index f1873ee..53d52c3 100644 --- a/openapi/vosi-tableset.yaml +++ b/openapi/vosi-tableset.yaml @@ -1,8 +1,8 @@ get: operationId: get-tableset - tag: VOSI - tag: table metadata - summary: VOSI Tables + tags: + - VOSI table metadata + summary: VOSI TableSet description: Provides the table metadata in the form of a TableSet description parameters: - name: detail @@ -14,7 +14,6 @@ get: enum: [min, max] responses: 200: - description: metadata for schemas and tables $ref: '#/components/schemas/tablesetDoc' 401: $ref: ./vosi-std-responses.yaml#/not-authenticated @@ -24,6 +23,7 @@ get: components: schemas: tablesetDoc: + description: metadata for schemas and tables content: text/xml: schema: From ffed326a05da2279ab8098af3a5b97f93069f00d Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Mon, 12 May 2025 09:00:51 -0700 Subject: [PATCH 12/15] update build workflow to v3 --- .github/workflows/build.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 7f015bf..d30fe10 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -21,7 +21,7 @@ jobs: steps: - name: Checkout the repository - uses: actions/checkout@v1 + uses: actions/checkout@v3 with: submodules: true From bab3bb169d96d6258cbb4fd09b87a74f80bf6d74 Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Mon, 12 May 2025 09:08:48 -0700 Subject: [PATCH 13/15] build: update upload-artifact to v4 preview: update checkout to v3 --- .github/workflows/build.yml | 2 +- .github/workflows/preview.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d30fe10..697cc76 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -39,7 +39,7 @@ jobs: test -f ${{ env.doc_name }}.bbl - name: Keep the PDF artefact - uses: actions/upload-artifact@v1 + uses: actions/upload-artifact@v4 with: name: PDF Preview path: ${{ env.doc_name }}.pdf diff --git a/.github/workflows/preview.yml b/.github/workflows/preview.yml index a410263..eb34a99 100644 --- a/.github/workflows/preview.yml +++ b/.github/workflows/preview.yml @@ -22,7 +22,7 @@ jobs: steps: - name: Checkout the repository - uses: actions/checkout@v1 + uses: actions/checkout@v3 with: submodules: true From 4100973b86478a72146c039ff358ce50aeb41723 Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Mon, 12 May 2025 10:15:11 -0700 Subject: [PATCH 14/15] finish draft text for user managed tables and add some TODOs --- VOSI.tex | 39 ++++++++++++++++++++++++++------- openapi/vosi-std-responses.yaml | 7 ++++++ openapi/vosi-table.yaml | 4 ++++ 3 files changed, 42 insertions(+), 8 deletions(-) diff --git a/VOSI.tex b/VOSI.tex index 526c7a4..5e5fe40 100644 --- a/VOSI.tex +++ b/VOSI.tex @@ -305,7 +305,7 @@ \subsection{Table metadata} specification (see ???). Here we give a brief summary of the main use and some examples but defer to the OpenAPI documents as the definitive specification. -\subsubsection{Tables} +\subsubsection{TableSet} The TableSet metadata is a hierarchical web resource with two levels of detail. Maximum detail (max) is the complete metadata including all details of the table structure (columns and foreign keys). Minimum detail (min) includes the full @@ -337,6 +337,7 @@ \subsubsection{Tables} respond with a usable TableSet document that has the maximum level of detail so may wish to always respond with the minimum level of detail. +\subsubsection{Table} The REST endpoint must also support a child resource for each table described in the TableSet document. The child resource must be named as it appears in the name of the corresponding child Table element. For @@ -345,8 +346,7 @@ \subsubsection{Tables} \begin{verbatim} GET http://example.net/srv/tables/ivoa.ObsCore \end{verbatim} - -would return a Table document describing the ivoa.ObsCore table in full +\noindent would return a Table document describing the ivoa.ObsCore table in full detail, starting: \begin{verbatim} @@ -355,6 +355,7 @@ \subsubsection{Tables} ... ... + \end{verbatim} The above \xmlel{vosi:table} document is the default response format; clients @@ -366,7 +367,7 @@ \subsubsection{User-managed Tables} of such tables and the authentication and authorization requirements for using this feature are an implementation detail that may vary from one service to another. At one extreme, users may be able to create tables anonymously and the table will persist for a short -perioid of time (days or weeks) before being removed. At the other extreme, users would +period of time (days or weeks) before being removed. At the other extreme, users would have to authenticate, have permission to create tables in a specific schema, and the tables would persist forever. @@ -377,24 +378,43 @@ \subsubsection{User-managed Tables} visible to and usable by other users (e.g. included in tables endpoint output and added to the \verb|tap_schema| of a TAP service). +The additional actions to allow users to manage tables are optional. The OpenAPI specification +says that a tables endpoint that does not support these actions must respond with an HTTP +405 status code; we expect that current services already respond this way. \todo{Do we need a +registry extension for VOSI-tables? just listing http verbs would be enough to avoid probing +for a 405 aka try and see if it works} + To create a new table: \begin{verbatim} PUT http://example.net/srv/tables/schema1.table1 \end{verbatim} -\noindent where the \verb|| payload can be either a VOSI-table or VOTable document. +\noindent where the \verb|| payload can be either a VOSI-table or VOTable document. For +a VOTable payload, the table may contain rows and those rows would be inserted into the created +table. To update table metadata: \begin{verbatim} POST http://example.net/srv/tables/schema1.table1 \end{verbatim} \noindent where only some of the table metadata can me modified (see below). In addition, if -the VOTable format is used for updating metadata, the VOTable must not contain any rows. +the VOTable format is used for updating metadata, the VOTable would not contain any rows (but +see below). To delete a table: \begin{verbatim} DELETE http://example.net/srv/tables/schema1.table1 \end{verbatim} +Some services may allow the caller to append additional rows to a table. This could be +implemented by accepting a table update that carries additional rows: +\begin{verbatim} +POST http://example.net/srv/tables/schema1.table1 +\end{verbatim} +\noindent where some metadata may be updated and rows in the input VOTable are appended +to the existing table. Other tabular data formats that are not sufficient for create may be +supported for appending rows: csv, tsv, fits, etc. \todo{Service should advertise the +list of acceptable formats} + \subsubsection{Mutable Table Metadata} Metadata that can be updated: \begin{itemize} @@ -402,14 +422,17 @@ \subsubsection{Mutable Table Metadata} \item column metadata that can be updated: utype, ucd, unit, description \end{itemize} -Column metadata where update would imply an structural change to the table (e.g. an SQL +Column metadata where update would imply a structural change to the table (e.g. an SQL alter table statement): \begin{itemize} \item column datatype, arraysize, xtype: alter column datatype? \item input table description has fewer columns: drop column? \item input table description has more columns: add column? +\item rename table +\item rename column(s) \end{itemize} -\noindent These are probably too fraught with pain in the details. +\noindent These are probably too fraught with pain in the details. \todo{Some CADC users have +asked for table and column renaming, which should be easy enough to implement} \subsection{Non-service metadata (non-normative)} diff --git a/openapi/vosi-std-responses.yaml b/openapi/vosi-std-responses.yaml index 040e498..853a6b6 100644 --- a/openapi/vosi-std-responses.yaml +++ b/openapi/vosi-std-responses.yaml @@ -40,3 +40,10 @@ not-implemented: text/plain: schema: type: string + +too-large: + description: the request content exceeds a server-side limitation on size + content: + text/plain: + schema: + type: string diff --git a/openapi/vosi-table.yaml b/openapi/vosi-table.yaml index 0589c17..6bbd09b 100644 --- a/openapi/vosi-table.yaml +++ b/openapi/vosi-table.yaml @@ -49,6 +49,8 @@ post: $ref: ./vosi-std-responses.yaml#/not-found 405: $ref: ./vosi-std-responses.yaml#/not-implemented + 413: + $ref: ./vosi-std-responses.yaml#/too-large put: operationId: create-table tags: @@ -78,6 +80,8 @@ put: $ref: ./vosi-std-responses.yaml#/not-found 405: $ref: ./vosi-std-responses.yaml#/not-implemented + 413: + $ref: ./vosi-std-responses.yaml#/too-large delete: operationId: delete-table tags: From 918ca72e4488367a332db306c302058229e9bd2c Mon Sep 17 00:00:00 2001 From: Patrick Dowler Date: Tue, 13 May 2025 12:59:04 -0700 Subject: [PATCH 15/15] add bad-request response for vosi-tableset (get) fix operationId for vosi-table (post) --- openapi/vosi-table.yaml | 2 +- openapi/vosi-tableset.yaml | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/openapi/vosi-table.yaml b/openapi/vosi-table.yaml index 6bbd09b..4c5b4c4 100644 --- a/openapi/vosi-table.yaml +++ b/openapi/vosi-table.yaml @@ -21,7 +21,7 @@ get: 404: $ref: ./vosi-std-responses.yaml#/not-found post: - operationId: update-table-metadata + operationId: update-table tags: - VOSI table metadata summary: proto - update table metadata and/or content (optional) diff --git a/openapi/vosi-tableset.yaml b/openapi/vosi-tableset.yaml index 53d52c3..4a27765 100644 --- a/openapi/vosi-tableset.yaml +++ b/openapi/vosi-tableset.yaml @@ -15,6 +15,8 @@ get: responses: 200: $ref: '#/components/schemas/tablesetDoc' + 400: + $ref: ./vosi-std-responses.yaml#/bad-request 401: $ref: ./vosi-std-responses.yaml#/not-authenticated 403: