Skip to content

Commit e673209

Browse files
garvinhickinggarvinhicking
committed
[TASK] Add basic XLIFF validation / linting
This check is implemented for the CI pipeline. It iterates all regular .xlf files in core extensions, ensures: - XLIFF is parseable XML - a supported XLIFF version (1.2, 2.0) is set (1.0 is dropped, not used in core anymore) Checks for XLIFF 1.2: -------------------- - matching namespaces (urn:oasis:names:tc:xliff:document:1.2) - attribute "file.source-language" is set to "en" - attribute "file.datatype" is set to "plaintext" - attribute "file.original" matches the own filename - "trans-unit" tags contain a non-empty ID - "trans-unit" does not contain deprecated keys (like mlang_labels_tablabel, mlang_labels_tabdescr, mlang_tabs_tab) without an "x-deprecated-since" attribute Checks for XLIFF 2.0: --------------------- - non-empty "file.id" attribute XLIFF 2.0 has less overhead, no more checks are implemented yet, and might be covered in a follow-up. As of now, only very few files use XLIFF 2.0. The concept could then be applied here, too. Local execution: --------------- The script is executed in the CI checks, but can also be called locally: ``` php Build/Scripts/checkIntegrityXliff.php ``` As a drive-by, the "originals" attributes of two xlf files are fixed, and one file converted from XLIFF 1.0 to 1.2. Also, the keys "description" and "short_description" are now enforced to at least have empty contents, because otherwise a fallback of the LanguageService would show their label names instead of a content. Note that checking xlf files of Site Sets is done seperately via `Build/Scripts/checkIntegritySetLabels.php`. Resolves: #108047 Related: #107790 Related: #108045 Releases: main Change-Id: If44816579deae764d097ad00497852db7b7079f0 Reviewed-on: https://review.typo3.org/c/Packages/TYPO3.CMS/+/91552 Tested-by: Lina Wolf <112@linawolf.de> Tested-by: Garvin Hicking <garvin@hick.ing> Reviewed-by: Benni Mack <benni@typo3.org> Reviewed-by: Lina Wolf <112@linawolf.de> Reviewed-by: Garvin Hicking <garvin@hick.ing> Tested-by: core-ci <typo3@b13.com> Tested-by: Benni Mack <benni@typo3.org>
1 parent 1d8d9fa commit e673209

File tree

24 files changed

+359
-4
lines changed

24 files changed

+359
-4
lines changed

‎Build/Scripts/checkIntegrityXliff.php‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
#!/usr/bin/env php
2+
<?php
3+
4+
declare(strict_types=1);
5+
6+
/*
7+
* This file is part of the TYPO3 CMS project.
8+
*
9+
* It is free software; you can redistribute it and/or modify it under
10+
* the terms of the GNU General Public License, either version 2
11+
* of the License, or any later version.
12+
*
13+
* For the full copyright and license information, please read the
14+
* LICENSE.txt file that was distributed with this source code.
15+
*
16+
* The TYPO3 project - inspiring people to share!
17+
*/
18+
19+
use Symfony\Component\Console\Formatter\OutputFormatter;
20+
use Symfony\Component\Console\Helper\Table;
21+
use Symfony\Component\Console\Helper\TableSeparator;
22+
use Symfony\Component\Console\Output\ConsoleOutput;
23+
use Symfony\Component\Finder\Finder;
24+
25+
require __DIR__ . '/../../vendor/autoload.php';
26+
27+
if (PHP_SAPI !== 'cli') {
28+
die('Script must be called from command line.' . chr(10));
29+
}
30+
31+
final readonly class CheckIntegrityXliff
32+
{
33+
private const expectedXliffDeprecations = [
34+
'mlang_labels_tablabel',
35+
'mlang_labels_tabdescr',
36+
'mlang_tabs_tab',
37+
];
38+
private const xliffModuleRegularExpression = '@Language/(module\.xlf|Modules/.+\.xlf)$@i';
39+
private const xliffModuleRequiredKeys = [
40+
'title',
41+
'description',
42+
'short_description',
43+
];
44+
private const XliffDeprecationKey = 'x-unused-since';
45+
46+
public function execute(): int
47+
{
48+
$filesToProcess = $this->findXliff();
49+
$output = new ConsoleOutput();
50+
$output->setFormatter(new OutputFormatter(true));
51+
52+
$testResults = [];
53+
$errors = [];
54+
/** @var \SplFileInfo $labelFile */
55+
foreach ($filesToProcess as $labelFile) {
56+
$fullFilePath = $labelFile->getRealPath();
57+
$result = $this->checkValidLabels($fullFilePath);
58+
if (isset($result['error'])) {
59+
$errors['EXT:' . $result['extensionKey'] . ':' . $result['shortLabelFile']] = $result['error'];
60+
}
61+
$testResults[] = $result;
62+
}
63+
64+
if ($testResults === []) {
65+
return 1;
66+
}
67+
68+
$table = new Table($output);
69+
$table->setHeaders([
70+
'EXT',
71+
'File',
72+
'Status',
73+
'Errorcode',
74+
]);
75+
foreach ($testResults as $result) {
76+
$table->addRow([
77+
$result['extensionKey'],
78+
$result['shortLabelFile'],
79+
(!isset($result['error']) ? "\xF0\x9F\x91\x8C" : "\xF0\x9F\x92\x80"),
80+
$result['errorcode'] ?? '',
81+
]);
82+
}
83+
$table->setFooterTitle(count($testResults) . ' files, ' . count($errors) . ' Errors');
84+
$table->render();
85+
86+
if ($errors === []) {
87+
return 0;
88+
}
89+
90+
$output->writeln('');
91+
$table = new Table($output);
92+
$table->setHeaders([
93+
'File',
94+
'Error',
95+
]);
96+
foreach ($errors as $file => $errorMessage) {
97+
$table->addRow([
98+
$file,
99+
$errorMessage,
100+
]);
101+
$table->addRow([
102+
new TableSeparator(),
103+
new TableSeparator(),
104+
]);
105+
}
106+
$table->setColumnMaxWidth(0, 40);
107+
$table->setColumnMaxWidth(1, 80);
108+
$table->render();
109+
110+
return 1;
111+
}
112+
113+
private function findXliff(): Finder
114+
{
115+
$finder = new Finder();
116+
return $finder
117+
->files()
118+
->in(__DIR__ . '/../../typo3/sysext/*/Resources/Private/Language/')
119+
->name('*.xlf');
120+
}
121+
122+
private function checkValidLabels(string $labelFile): array
123+
{
124+
$extensionKey = 'N/A';
125+
$shortLabelFile = basename($labelFile);
126+
if (preg_match('@sysext/(.+)/Resources/Private/Language/(.+)$@imsU', $labelFile, $matches)) {
127+
$extensionKey = $matches[1];
128+
$shortLabelFile = $matches[2];
129+
}
130+
131+
$result = [
132+
'shortLabelFile' => $shortLabelFile,
133+
'extensionKey' => $extensionKey,
134+
];
135+
136+
$xml = simplexml_load_file($labelFile);
137+
if ($xml === false) {
138+
$result['error'] = 'XML not parsable';
139+
$result['errorcode'] = 'XML';
140+
return $result;
141+
}
142+
143+
$attributes = (array)$xml->attributes();
144+
$version = $attributes['@attributes']['version'] ?? '';
145+
$supportedVersions = ['1.2', '2.0'];
146+
if (!in_array($version, $supportedVersions, true)) {
147+
$result['error'] = 'Incompatible version: ' . $version . ' (expected: ' . implode(', ', $supportedVersions) . ')';
148+
$result['errorcode'] = 'XLF version';
149+
return $result;
150+
}
151+
152+
$fileAttributes = (array)$xml->file->attributes();
153+
if ($version === '1.2') {
154+
$namespaces = $xml->getNamespaces(true);
155+
if (isset($namespaces[''])) {
156+
// Normalize empty namespace to "xml"
157+
$namespaces['xml'] = $namespaces[''];
158+
unset($namespaces['']);
159+
}
160+
$ns = 'urn:oasis:names:tc:xliff:document:1.2';
161+
if ($namespaces !== ['xml' => $ns]) {
162+
$result['error'] = 'Invalid XLIFF namespace: ' . json_encode($namespaces) . ' (expected: ' . $ns . ')';
163+
$result['errorcode'] = 'XML-NS';
164+
return $result;
165+
}
166+
$xml->registerXPathNamespace('x', $ns);
167+
168+
$sourceLanguage = $fileAttributes['@attributes']['source-language'] ?? '';
169+
$datatype = $fileAttributes['@attributes']['datatype'] ?? '';
170+
$original = $fileAttributes['@attributes']['original'] ?? '';
171+
$date = $fileAttributes['@attributes']['date'] ?? '';
172+
173+
$isIso = ($extensionKey === 'core' && str_starts_with($shortLabelFile, 'Iso/'));
174+
175+
if ($sourceLanguage !== 'en') {
176+
$result['error'] = 'Invalid source-language: ' . $sourceLanguage;
177+
$result['errorcode'] = 'file.source-language';
178+
return $result;
179+
}
180+
181+
if ($datatype !== 'plaintext') {
182+
$result['error'] = 'Invalid datatype: ' . $datatype;
183+
$result['errorcode'] = 'file.datatype';
184+
return $result;
185+
}
186+
187+
$expectedOriginals = [
188+
'EXT:' . $extensionKey . '/Resources/Private/Language/' . $shortLabelFile,
189+
'messages', // @todo is this right?
190+
];
191+
192+
if ($isIso) {
193+
$expectedOriginals[] = 'EXT:core/Resources/Private/Language/countries.xlf';
194+
}
195+
if (!in_array($original, $expectedOriginals, true)) {
196+
$result['error'] = 'Invalid original: ' . $original . ' (expected: ' . implode(', ', $expectedOriginals) . ')';
197+
$result['errorcode'] = 'file.original';
198+
return $result;
199+
}
200+
201+
if (!$isIso && (strtotime($date) === false || strtotime($date) === 0)) {
202+
$result['error'] = 'Invalid date: ' . $date;
203+
$result['errorcode'] = 'file.date';
204+
return $result;
205+
}
206+
207+
// verify these are deprecated:
208+
$transUnits = $xml->xpath('/x:xliff/x:file/x:body/x:trans-unit');
209+
$seenKeys = [];
210+
foreach ($transUnits as $unit) {
211+
$unitAttributes = (array)$unit;
212+
$unitId = $unitAttributes['@attributes']['id'] ?? '';
213+
if ($unitId === '') {
214+
$result['error'] = 'TransUnit without ID specified.';
215+
$result['errorcode'] = 'trans-unit';
216+
return $result;
217+
}
218+
219+
if (in_array($unitId, self::expectedXliffDeprecations, true)
220+
&& ($unitAttributes['@attributes'][self::XliffDeprecationKey] ?? '') === ''
221+
) {
222+
$result['error'] = 'TransUnit ' . $unitId . ' missing ' . self::XliffDeprecationKey . ' attribute.';
223+
$result['errorcode'] = 'trans-unit.' . self::XliffDeprecationKey;
224+
return $result;
225+
}
226+
$seenKeys[$unitId] = $unitId;
227+
}
228+
229+
if (preg_match(self::xliffModuleRegularExpression, $labelFile)) {
230+
// Hit on any "backend module file".
231+
foreach (self::xliffModuleRequiredKeys as $requiredKey) {
232+
if (!isset($seenKeys[$requiredKey])) {
233+
$result['error'] = 'Backend module missing label ' . $requiredKey . '.';
234+
$result['errorcode'] = 'missing ' . $requiredKey;
235+
return $result;
236+
}
237+
}
238+
}
239+
} else {
240+
$fileId = $fileAttributes['@attributes']['id'] ?? '';
241+
if ($fileId === '') {
242+
$result['error'] = 'Missing file.id';
243+
$result['errorcode'] = 'file.id';
244+
return $result;
245+
}
246+
247+
// XLIFF 2.0 has no deprecation syntax check yet.
248+
}
249+
250+
return $result;
251+
}
252+
}
253+
254+
exit((new CheckIntegrityXliff())->execute());

‎Build/Scripts/runTests.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -943,6 +943,10 @@ case ${TEST_SUITE} in
943943
${CONTAINER_BIN} run ${CONTAINER_COMMON_PARAMS} --name check-integrity-set-labels-${SUFFIX} ${IMAGE_PHP} php -dxdebug.mode=off Build/Scripts/checkIntegritySetLabels.php
944944
SUITE_EXIT_CODE=$?
945945
;;
946+
checkIntegrityXliff)
947+
${CONTAINER_BIN} run ${CONTAINER_COMMON_PARAMS} --name check-integrity-set-labels-${SUFFIX} ${IMAGE_PHP} php -dxdebug.mode=off Build/Scripts/checkIntegrityXliff.php
948+
SUITE_EXIT_CODE=$?
949+
;;
946950
checkComposer)
947951
${CONTAINER_BIN} run ${CONTAINER_COMMON_PARAMS} --name check-composer-${SUFFIX} ${IMAGE_PHP} php -dxdebug.mode=off Build/Scripts/checkIntegrityComposer.php
948952
SUITE_EXIT_CODE=$?

‎Build/gitlab-ci/nightly/integrity.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ integration various php 8.5:
3030
- Build/Scripts/runTests.sh -s checkComposer -p 8.5
3131
- Build/Scripts/runTests.sh -s checkIntegrityPhp -p 8.5
3232
- Build/Scripts/runTests.sh -s checkIntegritySetLabels -p 8.5
33+
- Build/Scripts/runTests.sh -s checkIntegrityXliff -p 8.5
3334
- Build/Scripts/runTests.sh -s lintServicesYaml -p 8.5
3435
- Build/Scripts/runTests.sh -s lintYaml -p 8.5
3536
- Build/Scripts/runTests.sh -s checkFilesAndPathsForSpaces -p 8.5

‎Build/gitlab-ci/pre-merge/integrity.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ integration various php 8.2 pre-merge:
3232
- Build/Scripts/runTests.sh -s checkComposer -p 8.2
3333
- Build/Scripts/runTests.sh -s checkIntegrityPhp -p 8.2
3434
- Build/Scripts/runTests.sh -s checkIntegritySetLabels -p 8.2
35+
- Build/Scripts/runTests.sh -s checkIntegrityXliff -p 8.5
3536
- Build/Scripts/runTests.sh -s lintServicesYaml -p 8.2
3637
- Build/Scripts/runTests.sh -s lintYaml -p 8.2
3738
- Build/Scripts/runTests.sh -s checkFilesAndPathsForSpaces -p 8.2

‎typo3/sysext/backend/Resources/Private/Language/Modules/pagetsconfig.xlf‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@
66
<trans-unit id="title">
77
<source>Page TSconfig</source>
88
</trans-unit>
9+
<!-- intentionally left blank, not utilized (for now) -->
10+
<trans-unit id="short_description">
11+
<source></source>
12+
</trans-unit>
13+
<trans-unit id="description">
14+
<source></source>
15+
</trans-unit>
916
</body>
1017
</file>
1118
</xliff>

‎typo3/sysext/backend/Resources/Private/Language/Modules/pagetsconfig_active.xlf‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@
66
<trans-unit id="title">
77
<source>Active page TSconfig</source>
88
</trans-unit>
9+
<!-- intentionally left blank, not utilized (for now) -->
10+
<trans-unit id="short_description">
11+
<source></source>
12+
</trans-unit>
13+
<trans-unit id="description">
14+
<source></source>
15+
</trans-unit>
916
</body>
1017
</file>
1118
</xliff>

‎typo3/sysext/backend/Resources/Private/Language/Modules/pagetsconfig_includes.xlf‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@
66
<trans-unit id="title">
77
<source>Included page TSconfig</source>
88
</trans-unit>
9+
<!-- intentionally left blank, not utilized (for now) -->
10+
<trans-unit id="short_description">
11+
<source></source>
12+
</trans-unit>
13+
<trans-unit id="description">
14+
<source></source>
15+
</trans-unit>
916
</body>
1017
</file>
1118
</xliff>

‎typo3/sysext/backend/Resources/Private/Language/Modules/pagetsconfig_pages.xlf‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,13 @@
66
<trans-unit id="title">
77
<source>Pages containing page TSconfig</source>
88
</trans-unit>
9+
<!-- intentionally left blank, not utilized (for now) -->
10+
<trans-unit id="short_description">
11+
<source></source>
12+
</trans-unit>
13+
<trans-unit id="description">
14+
<source></source>
15+
</trans-unit>
916
</body>
1017
</file>
1118
</xliff>

‎typo3/sysext/backend/Resources/Private/Language/Wizards/move_content_elements.xlf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<xliff version="1.2" xmlns="urn:oasis:names:tc:xliff:document:1.2">
3-
<file source-language="en" datatype="plaintext" original="EXT:core/Resources/Private/Language/Wizards/move_page.xlf" date="2024-03-08T20:22:34Z" product-name="backend">
3+
<file source-language="en" datatype="plaintext" original="EXT:backend/Resources/Private/Language/Wizards/move_content_elements.xlf" date="2024-03-08T20:22:34Z" product-name="backend">
44
<header/>
55
<body>
66
<trans-unit id="headline.move">

‎typo3/sysext/backend/Resources/Private/Language/Wizards/move_page.xlf‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
<?xml version="1.0" encoding="UTF-8"?>
22
<xliff version="1.2" xmlns="urn:oasis:names:tc:xliff:document:1.2">
3-
<file source-language="en" datatype="plaintext" original="EXT:core/Resources/Private/Language/Wizards/move_page.xlf" date="2024-03-08T20:22:34Z" product-name="backend">
3+
<file source-language="en" datatype="plaintext" original="EXT:backend/Resources/Private/Language/Wizards/move_page.xlf" date="2024-03-08T20:22:34Z" product-name="backend">
44
<header/>
55
<body>
66
<trans-unit id="headline">

0 commit comments

Comments
 (0)