Adapt CSV output format to match with SUISA Tarif S (2020-2023)

[paraenggu]

Overview

The CSV output shall be adapted to match with the SUISA document Gemeinsamer Tarif S 2020 – 2023, as published on the SUISA Radio and TV page.

Background

The following parts are relevant from the Gemeinsamer Tarif S 2020 – 2023 document:

G. Verzeichnisse (click to expand)

G. Verzeichnisse

32 Die Sender melden die unter Buchstabe G genannten Programmangaben in elektronischer Form in einem standardisierten importierbaren Format.

a) Radio

33 Die Sender melden der SUISA die in ihren Programmen gesendeten Musik bzw. Ton- und Tonbildträger, einschliesslich der Musikteppiche und Jingles. Die Meldungen sind gemäss Anhang I zu diesem Tarif vorzulegen.

34 Die Angaben enthalten

• Titel des Musikwerks
• Name des Komponisten
• Name des bzw. der Hauptinterpreten
• Label
• ISRC der benützten Aufnahme (Eine Pflicht zur Meldung und Dokumentation des ISRC besteht mindestens dann, wenn der ISRC zusammen mit der Aufnahme vom Lieferanten der Aufnahme in irgendeiner Form mitgeteilt bzw. mitgeliefert wird oder nachträglich unter Hinweis auf eine bestimmte Aufnahme nachgemeldet wird. Nachmeldungen und Korrekturen von ISRC hat der Sender sofort zu verarbeiten und der SUISA mitzuteilen.)
• vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer
• Sendezeit
• Sendedauer

35 [...]

36 Sofern ein Sender nicht in der Lage ist, eventuelle Überblendungen zwischen zwei gleichzeitig gesendeten Musikwerken oder zwischen Musik und gesprochenem Text (während denen Musik gespielt wird, diese jedoch für die Hörer nicht wahrnehmbar ist) in seinen Meldungen anzugeben, tolerieren SUISA und SWISSPERFORM eine Abweichung von 5 % zwischen der vom Sender angegebenen Gesamtdauer der geschützten Musik bzw. der geschützten Handelstonträger gemäss der Meldungen nach Ziffer 34 und dem vom Sender deklarierten Musikanteil bzw. Anteil gesendeter Handelstonträger an der Sendezeit seiner Programme.

Diese Toleranz von 5 % gilt nur, soweit die vom Sender gelieferten Daten vollständige Angaben über die gesendete Musik und die gesendeten Tonträger sowie den jeweiligen Sendedauern enthalten und soweit keine geeignetere Methode zur Verfügung steht, mit der die Überblendungen ermittelt werden können.

[...]

c) Gemeinsame Bestimmungen

44 Die von anderen Sendern regelmässig übernommenen Programme sind mit den folgenden Angaben der SUISA mitzuteilen

• Name des Senders
• Zahl der Sendestunden der übernommenen Programme.

d) Termine

45 Alle Angaben sind der SUISA – sofern vorstehend nicht anders bestimmt – monatlich jeweils bis zum Ende des folgenden Monats zuzustellen. Die Angaben zu den gesendeten Jingles können quartalsweise zugestellt werden.

Anhang I - Vorlage Formatierung Sendelisten Radio (click to expand)

The referenced Anhang I (appendix I) within paragraph 33, named "Vorlage Formatierung Sendelisten Radio", contains the fields in the following order:

• Titel
• Komponist
• Interpret
• Interpreten-Info
• Sender
• Sendedatum
• Sendedauer
• Sendezeit
• Werkverzeichnisangaben
• ISRC
• Label
• CD ID / Katalog-Nummer
• Aufnahmedatum
• Aufnahmeland
• Erstveröffentlichungsdatum
• Titel des Tonträgers (Albumtitel)
• Autor Text
• Track Nummer
• Genre
• Programm
• Bestellnummer
• Marke
• Label Code
• EAN/GTIN

interestingly it lacks a field corresponding to "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" according to paragraph 34.

There's an additional template called Vorlage für Sendemeldungen: Gemeinsamer Tarif S:

• This template has the same fields as the Anhang I mentioned within paragraph 33, but includes an additional field named Identifikationsnummer which matches with the mentioned data "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" of paragraph 34, which is missing in Anhang I.
• The template seems to uses a coloring scheme (gray and white), whereas the grey fields match with the required data mentioned in paragraph 34 (except for Label Code, which is probably only a formatting error, as it's not mentioned in paragraph 34 although Label and Label Codeare marked as mandatory, only one is required according to clarifications with SIUSA as of 2023-02)

Hypothesis: The data under paragraph 34 is mandatory, whereas all the other data can be considered optional.

This would imply that at least the following fields with the names below, have to be provided:

• Titel: Titel sind in Orginalsprache gemäss Tonträger, inklusive ggf. Versionsabgaben ("live in London" "dirty remix" etc.) anzugeben
• Komponist: Name des/der Komponisten
• Interpret: Name des (Haupt-) Interpreten, bzw. Gruppenname
• Sendedatum: Datum der Nutzung im Einheitformat, z.B. YYYYMMDD - 20110101
• Sendedauer: Dauer der Nutzung im Einheitsformat, z.B. hh:mm:ss - 00:03:54
• Sendezeit: Startzeitpunkt der Nutzung im Einheitsformat, z.B. hh:mm:ss - 06:09:13
• ISRC: ISRC Code des Quelltonträgers
• Label: Name des Labels
• Identifikationsnummer: Vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer

The header of the current CSV output looks as follows and currently doesn't exactly match with the fields from the template:

Sendedatum,Sendezeit,Sendedauer,Titel,Künstler,Komponist,ISRC,Label

Proposal

To match with the template from the Gemeinsamer Tarif S 2020 – 2023 document the CSV header and field ordering shall be adapted as follows:

Titel,Komponist,Interpret,Sendedatum,Sendedauer,Sendezeit,ISRC,Label,Identifikationsnummer

• Name the fields (within the CSV header row) exactly the same as within the template:
  • Künstler 👉 Interpret
• Use the same field ordering as within the template
• For the Identifikationsnummer the PR feat: add CRID as Identifikationsnummer #185 could be used
• Ensure that additional fields (with possible empty values) could be introduced (depending on the outcome of the clarifications mentioned below)

Additionally, the date format of the Sendedatum shall be changed from DD/MM/YY to YYYYMMDD in order to match with the example from the template.

The name of the CSV file shall be changed from suisa_sendemeldung_<MONTH>.csv to <RADIO-STATION>_<YYYY>_<MM>.csv (e.g. my-station_2023_01.csv), in order to match with the naming convention of Anhang I.

Preliminary clarifications

Before the change can be made, the following preliminary clarifications should be made with SUISA:

• Is the Gemeinsamer Tarif S 2020 – 2023 document still valid for 2023, or is there a newer superseding one?

  Gültigkeitsdauer (click to expand)

  49 Dieser Tarif ist vom 1. Januar 2020 bis zum 31. Dezember 2022 gültig.
  [...]
  50 Die Gültigkeitsdauer des Tarifs verlängert sich automatisch um jeweils ein Jahr bis längstens 31. Dezember 2025, wenn er nicht von einem der Verhandlungspartner durch schriftliche Anzeige an den anderen ein Jahr vor Ablauf gekündigt wird.

  Feedback from SUISA as of 2023-02:
  Yes the Tarif is still valid for 2023 as confirmed by SUISA on 2023-02-02

• Is the assumption correct, that the fields corresponding to the data mentioned in paragraph 34 are mandatory and everything else is optional?

  Feedback from SUISA as of 2023-02:
  Yes, that's correct. The following note applies to the fields Label and Label Code: Only one of them must be provided, that's why both cells are marked as mandatory (grey background) in Vorlage für Sendemeldungen: Gemeinsamer Tarif S

• Why is the there no field matching the data "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" reference within paragraph 34, in Anhang I?

  Feedback from SUISA as of 2023-02:
  The field has to be provided and is named Identifikationsnummer as assumed

• Does SUISA expect the report to contain all fields from the template or only the mandatory ones?

  Feedback from SUISA as of 2023-02:
  Yes, all fields are expected, but only values for the mandatory ones must be provided, the format and field ordering must be based on the Vorlage für Sendemeldungen: Gemeinsamer Tarif S

• What value is expected in case a certain field is not applicable ("Sofern zutreffend/bekannt"), e.g. empty value or N/A?

  Feedback from SUISA as of 2023-02:
  An empty value is expected.

• Is it still OK to deliver the report as CSV or is an XLSX format (same as the template) required?

  Feedback from SUISA as of 2023-02:
  SUISA expects the report in XLSX format for 2023 based on the Vorlage für Sendemeldungen: Gemeinsamer Tarif S and not CSV anymore, addressed in Switch from CSV to Excel format for reports #203

• Request approved example entries for multiple genres (rock/pop, classic, jazz and electronics as well as jingles)

  Feedback from SUISA as of 2023-02:
  Example entries have been provided.

[hairmare]

I'm already on a large PR to take care of a lot of this.

What value is expected in case a certain field is not applicable ("Sofern zutreffend/bekannt"), e.g. empty value or N/A?

I have been assuming that we leave Null values where we don't have infos empty as i would assume from any sane standard like RFC 4180.

Request approved example entries for multiple genres (rock/pop, classic, jazz and electronics as well as jingles)

I can dig up official lists from sources like EBU/ETSI, but we don't currently have any such dava available from acr cloud so i wouldn't dig too deep

All the other points are pretty much what i also discovered, but from a purely technical perspective probably ignoreable-ish.

Like i said, been hacking... what my working copy has generates what is now in #188 and this CSV (with a filename of rabe_2022_12.csv):

Titel,Komponist,Interpret,Interpreten-Info,Sender,Sendedatum,Sendedauer,Sendezeit,Werkverzeichnisangaben,ISRC,Label,CD ID / Katalog-Nummer,Aufnahmedatum,Aufnahmeland,Erstveröffentlichungsdatum,Titel des Tonträgers (Albumtitel),Autor Text,Track Nummer,Genre,Programm,Bestellnummer,Marke,Label Code,EAN/GTIN,vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer
Penumbra,,Inexedra,,Radio Bern RaBe,20230101,0:03:58,00:07:22,,QZDA81853826,700103 Records DK,,,,20180831,Lexicon,,,,,,,,840090528844,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T000722.00Z&acrid=6e655c4481f02512e00f76ef92b53fa2
Navi,,A94,,Radio Bern RaBe,20230101,0:00:10,00:11:31,,TCAFM2103487,A94,,,,20210326,Navi,,,,,,,,859746613438,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T001131.00Z&acrid=1dc420d7141d3509662002dceeef4adb
Manœuvre,"Felix Leiter, Mark Maitland",Felix Leiter,,Radio Bern RaBe,20230101,0:00:06,00:11:42,,FRX201561956,BELIEVE - Dice Records,,,,2015,Manœuvre,,,,,,,,3614592171474,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T001142.00Z&acrid=fff5fd73f45ccb190a9d0948815f702a
Penumbra,,Inexedra,,Radio Bern RaBe,20230101,0:02:13,00:11:51,,QZDA81853826,700103 Records DK,,,,20180831,Lexicon,,,,,,,,840090528844,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T001151.00Z&acrid=6e655c4481f02512e00f76ef92b53fa2
Unique (Original Mix),,Acid Drink,,Radio Bern RaBe,20230101,0:00:46,00:14:14,,GBKQU1346332,Digital Empire Records,,,,20130318,Unique,,,,,,,,5052653539098,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T001414.00Z&acrid=48836c1b08e0b617b5fe01b5598ce670
Round the World,,LukHash,,Radio Bern RaBe,20230101,0:02:15,00:15:01,,USHM91658174,Lukhash,,,,20170113,Glitch,,,,,,,,191061170341,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T001501.00Z&acrid=fd5a35a6c14699ea1792f266573e5d6c
Touch the Sky,,Isidor,,Radio Bern RaBe,20230101,0:04:36,00:17:15,,QZAPG1746964,BELIEVE - Kiez Beats,,,,20200424,"Synthwave, Vol. 7",,,,,,,,3616404272621,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T001715.00Z&acrid=c36c859f6afb92fe5bdd71dbcf37e560
Los je dan,,Max Vincent,,Radio Bern RaBe,20230101,0:03:14,00:22:02,,NOAPI1692427,ORCHARD - Discom,,,,20160815,The Future Has Designed Us,,,,,,,,7071245304898,crid://rabe.ch/v1/clubtour-b%C3%B6lz-no-eis-synthvester:-s%C3%A4gegasse-burgdorf#t=clock=20230101T002202.00Z&acrid=83cc41236be09beb9f2227c4264a2891

I'm at the point where i'm finalising the thing for a PR, so we'll be able to reason over this in code soon :)

Main difference to this proposal is that the field "Identifikationsnummer" is called "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" in my implementation. The implementation adds all fields, and fills them in where i noticed we have data available if possible but the main focus is still on those mentions in "Buchstabe G" explicitly. I also stop guessing that we could just do artist = composer if no compose data is available.

[hairmare]

Is the Gemeinsamer Tarif S 2020 – 2023 document still valid for 2023, or is there a newer superseding one?

I would assert the document is indeed still valid, if not we would have gotten a letter a year ago (ein Jahr vor Ablauf). The fact that it's still what we find via their webinterface and we were told to "look at the tariff" compounds this.

Looking forward, according to "automatisch um jeweils ein Jahr bis längstens 31. Dezember 2025" we either get a letter on 31. Dec 2023 (this year) or we are in default clause territory anyway and there will most likely be a new "Gemeinsamer Tarif S 2025 – 2028" that we would then assume to be valid through to 2030.

For these reasons imma mark this preliminary clarification as clarified.

[hairmare]

Is the assumption correct, that the fields corresponding to the data mentioned in paragraph 34 are mandatory and everything else is optional?

my understanding is that there are no clear references to the "Vorlage" at the end of the PDF in the legalese part of the document. this is further compounded that the only real way to reference the "Vorlage" is with unprinted page numbers and there are no chapter numbers that could be referenced ala "Buchstabe X". not true, imma reference it as "Anhang 1"

IMO the legalese sounding part is what Eidgenössische Schiedskommission für die Verwertung von Urheber-
rechten und verwandten Schutzrechten approved and the last few pages seem to be an example, esp. given that "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" is missing there but clearly mandated in the legal bits.

[hairmare]

Why is the there no field matching the data "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" reference within paragraph 34, in Anhang I?

Yes, why tf is it missing, do they even schiedskommission over in the dept of schiedskomissioning

[hairmare]

Is it still OK to deliver the report as CSV or is an XLSX format (same as the template) required?

"Anhang 1" has us covered:

lol, the xlsx file downloadable from here only has "Name des Files: SENDER_JAHR_MONAT" in that comment, but it's also named "Vorlage_fuer_Sendemeldungen__GTS__2020-2022.xlsx" for whatever reasons.

[paraenggu]

The idea was to clarify the topics officially with SUISA instead of assuming that the interpretation will be correct, this didn't worked out in the past.

The description of the issue has all relevant parts from the documents and templates.

[hairmare]

The idea was to clarify the topics officially with SUISA instead of assuming that the interpretation will be correct

The current implementation was not based on assumptions but rather on clarifying the topics officially with SUISA. It's rather clear that the staff at SUISA said it's ok when that was clearly not the case in the past...

anyways, an initial implementation that addresses this is in #189 and can be finalized if we get any legally sound feedback directly.

please let me know in time (before feb) if we need to defer sending this months report to a later date. We are required to send it in the following month and currently do so on the first business day of the sais month, in theory this means we can defer sending it until the end of Feb (i'm not proposing end of month as new standard here, this would just be for the Jan 2023 report).

[paraenggu]

Update after clarifications with SUISA as of 2023-02

Is the Gemeinsamer Tarif S 2020 – 2023 document still valid for 2023, or is there a newer superseding one?

Yes the Tarif is still valid for 2023 as confirmed by SUISA on 2023-02-02

Why is the there no field matching the data "vom Sender der Aufnahme selbst zugewiesene Identifikationsnummer" reference within paragraph 34, in Anhang I?

The field has to be provided and is named Identifikationsnummer as assumed

Does SUISA expect the report to contain all fields from the template or only the mandatory ones?

Yes, all fields are expected, but only values for the mandatory ones must be provided, the format and field ordering must be based on the Vorlage für Sendemeldungen: Gemeinsamer Tarif S

Is it still OK to deliver the report as CSV or is an XLSX format (same as the template) required?

SUISA expects the report in XLSX format for 2023 based on the Vorlage für Sendemeldungen: Gemeinsamer Tarif S and not CSV anymore, addressed in #203

[hairmare]

btw, the International Standard Recording Code (ISRC) Handbook – 4th Edition, 2021 has this on compliant ISRC codes (emphasis mine):

4.7 Reference Metadata
When ISRC is assigned, the ‘binding’ of the ISRC to the specific recording is secured through
‘reference metadata’. Reference metadata (sometimes called “minimum metadata” or “kernel
metadata”) is a small set of descriptive data such that if two recordings are different, the
reference metadata will differ in at least one field. The reference metadata that must be held
alongside each ISRC is the recording’s:
▪ Title
▪ Version title
▪ Artist or band name principally associated with the recording (i.e. main artist name)
▪ Duration of the recording
▪ Type (i.e. sound recording or music video)
▪ Year of first publication
The full specification of this reference metadata is contained in Annex E.
Any assignment of ISRC should be accompanied by the storage, in the Registrant’s systems at
least, of this dataset alongside the ISRC. ISRCs which lack this data are not fully compliant under
the ISRC standard.
To obtain the greatest benefit from the ISRC system, Registrants should ensure that details of
the recording and its ISRC are supplied to all relevant repertoire databases.

One could construe that the reference metadata fields should thus be considered optional from our reporting perspective in cases where we provide an ISRC.

An argument that could be made revolves around privacy, the way that ISRC codes are structured it would very well be possible to report on tracks so they get assigned properly to a label without more detail. imo, it shouldn't be up to us to decide what title or name is associated with any given ISRC as this information might be privy to only the registrant and still be compliant with the spec the handbook describes.

I assume that suisa will have contracts in place that require an isrc registrant to provide them with metadata as they are a relevant repertoire database, but this is just me assuming. I don't think we are part of that agreement and, security wise, it would be nice if we didn't have to handle data that isn't ours (esp. given new data privacy laws and the fact that the artist field might contain pii).

anyways, maybe a privacy centric argument can help our case. also, the handbook i linked is an interesting read

[paraenggu]

Update after clarifications with SUISA as of 2023-02

Is the assumption correct, that the fields corresponding to the data mentioned in paragraph 34 are mandatory and everything else is optional?

Yes, that's correct. Note regarding the fields Label and Label Code: Only one of them must be provided, that's why both cells are marked as mandatory (grey background) in Vorlage für Sendemeldungen: Gemeinsamer Tarif S:

What value is expected in case a certain field is not applicable ("Sofern zutreffend/bekannt"), e.g. empty value or N/A?

An empty value is expected.