artifacts offline install - implausibly old time stamp 1970-01-01 00:00:00 #13415

fgelcer · 2023-04-02T19:42:20Z

Issue description

This issue is a regression.
It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

installation of ScyllaDB on a fresh CentOS 7 failed (it may actually succeeded, but the test stopped because of the errors encountered during the installation).

2023-03-31 05:56:39.295: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=507e9bf6-1e7d-4688-8718-7032e7d0c50a, source=ArtifactsTest.SetUp()
exception=[Node artifacts-centos7-jenkins-db-node-804b2316-0-1 [104.196.128.187 | 10.142.0.39] (seed: True)] NodeSetupFailed: Encountered a bad command exit code!
Command: 'sudo bash -cxe "\ntar xvfz ./unified_package.tar.gz\ncd ./scylla-*\n./install.sh --housekeeping\ncd -\nrm -f /tmp/scylla.yaml\n"'
Exit code: 1
Stdout:
fs.nr_open = 1073741816
fs.inotify.max_user_instances = 1200
kernel.sched_tunable_scaling = 0
kernel.sched_min_granularity_ns = 500000
kernel.sched_wakeup_granularity_ns = 450000
kernel.sched_latency_ns = 1000000
kernel.sched_autogroup_enabled = 0
kernel.numa_balancing = 0
vm.swappiness = 1
Scylla offline install completed.
Stderr:
+ tar xvfz ./unified_package.tar.gz
tar: scylla-5.3.0~dev/scylla-python3/bin/python3.11: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/python3: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/geomet: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/traceback-with-variables: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/scylla-api-client: implausibly old time stamp 1970-01-01 00:00:00
+ cd ./scylla-5.3.0~dev
+ ./install.sh --housekeeping
Traceback (most recent call last):
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/cluster.py", line 3640, in node_setup
cl_inst.node_setup(_node, **setup_kwargs)
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/cluster.py", line 4291, in node_setup
self._scylla_install(node)
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/cluster.py", line 4370, in _scylla_install
node.offline_install_scylla(unified_package=self.params.get('unified_package'),
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/cluster.py", line 2062, in offline_install_scylla
self.remoter.run('sudo bash -cxe "%s"' % install_cmds)
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/remote/remote_base.py", line 613, in run
result = _run()
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/utils/decorators.py", line 67, in inner
return func(*args, **kwargs)
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/remote/remote_base.py", line 604, in _run
return self._run_execute(cmd, timeout, ignore_status, verbose, new_session, watchers)
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/remote/remote_base.py", line 537, in _run_execute
result = connection.run(**command_kwargs)
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 620, in run
return self._complete_run(channel, exception, timeout_reached, timeout, result, warn, stdout, stderr)
File "/tmp/workspace/scylla-master/artifacts-offline-install/artifacts-centos7-test/scylla-cluster-tests/sdcm/remote/libssh2_client/__init__.py", line 654, in _complete_run
raise UnexpectedExit(result)
sdcm.remote.libssh2_client.exceptions.UnexpectedExit: Encountered a bad command exit code!
Command: 'sudo bash -cxe "\ntar xvfz ./unified_package.tar.gz\ncd ./scylla-*\n./install.sh --housekeeping\ncd -\nrm -f /tmp/scylla.yaml\n"'
Exit code: 1
Stdout:
fs.nr_open = 1073741816
fs.inotify.max_user_instances = 1200
kernel.sched_tunable_scaling = 0
kernel.sched_min_granularity_ns = 500000
kernel.sched_wakeup_granularity_ns = 450000
kernel.sched_latency_ns = 1000000
kernel.sched_autogroup_enabled = 0
kernel.numa_balancing = 0
vm.swappiness = 1
Scylla offline install completed.
Stderr:
+ tar xvfz ./unified_package.tar.gz
tar: scylla-5.3.0~dev/scylla-python3/bin/python3.11: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/python3: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/geomet: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/traceback-with-variables: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/scylla-api-client: implausibly old time stamp 1970-01-01 00:00:00
+ cd ./scylla-5.3.0~dev
+ ./install.sh --housekeeping

How frequently does it reproduce?

last 2 runs failed with this error

Installation details

Cluster size: 1 nodes (n1-standard-2)

Scylla Nodes used in this run:

artifacts-centos7-jenkins-db-node-804b2316-0-1 (104.196.128.187 | 10.142.0.39) (shards: 2)

OS / Image: https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/family/centos-7 (gce: us-east1)

Test: artifacts-centos7-test
Test id: 804b2316-eb68-46f6-a2a5-f20aba954834
Test name: scylla-master/artifacts-offline-install/artifacts-centos7-test
Test config file(s):

centos7.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 804b2316-eb68-46f6-a2a5-f20aba954834
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 804b2316-eb68-46f6-a2a5-f20aba954834

Logs:

db-cluster-804b2316.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/804b2316-eb68-46f6-a2a5-f20aba954834/20230331_055758/db-cluster-804b2316.tar.gz
sct-runner-events-804b2316.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/804b2316-eb68-46f6-a2a5-f20aba954834/20230331_055758/sct-runner-events-804b2316.tar.gz
sct-804b2316.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/804b2316-eb68-46f6-a2a5-f20aba954834/20230331_055758/sct-804b2316.log.tar.gz

Jenkins job URL

The text was updated successfully, but these errors were encountered:

fgelcer · 2023-04-02T19:42:33Z

@yaronkaikov

yaronkaikov · 2023-04-02T21:01:42Z

@syuu1228 Can you please have a look?

syuu1228 · 2023-04-03T09:40:23Z

The unified package url was https://downloads.scylladb.com/unstable/scylla/master/relocatable/2023-03-31T03:19:03Z/scylla-unified-5.3.0~dev-0.20230331.160c184d0b0f.x86_64.tar.gz, I checked timestamp of the tar file, it contains incorrect timestamp files as reported:

$ tar tvpf scylla-unified-5.3.0~dev-0.20230331.160c184d0b0f.x86_64.tar.gz |grep scylla-python3/bin
drwxr-xr-x jenkins/jenkins        0 2023-03-31 10:25 scylla-5.3.0~dev/scylla-python3/bin/
-rwxr-xr-x jenkins/jenkins      275 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/python3.11
lrwxrwxrwx jenkins/jenkins        0 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/python3 -> python3.11
-rwxr-xr-x jenkins/jenkins      212 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/geomet
-rwxr-xr-x jenkins/jenkins     1047 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/traceback-with-variables
-rwxr-xr-x jenkins/jenkins      229 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/scylla-api-client

I also checked locally built unified package, the result was same:

$ tar tvpf build/release/dist/tar/scylla-unified-5.3.0~dev-0.20230402.f2cef98e7c13.x86_64.tar.gz |grep scylla-python3/bin
drwxr-xr-x syuu/docker        0 2023-04-02 18:25 scylla-5.3.0~dev/scylla-python3/bin/
-rwxr-xr-x syuu/docker      275 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/python3.11
lrwxrwxrwx syuu/docker        0 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/python3 -> python3.11
-rwxr-xr-x syuu/docker      212 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/geomet
-rwxr-xr-x syuu/docker     1047 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/traceback-with-variables
-rwxr-xr-x syuu/docker      229 1970-01-01 09:00 scylla-5.3.0~dev/scylla-python3/bin/scylla-api-client
$ tar tvpf build/release/dist/tar/scylla-python3-5.3.0~dev-0.20230402.f2cef98e7c13.x86_64.tar.gz |grep scylla-python3/bin
-rwxr-xr-x 0/0             275 1970-01-01 09:00 scylla-python3/bin/python3.11
lrw-r--r-- 0/0               0 1970-01-01 09:00 scylla-python3/bin/python3 -> python3.11
-rwxr-xr-x 0/0             212 1970-01-01 09:00 scylla-python3/bin/geomet
-rwxr-xr-x 0/0            1047 1970-01-01 09:00 scylla-python3/bin/traceback-with-variables
-rwxr-xr-x 0/0             229 1970-01-01 09:00 scylla-python3/bin/scylla-api-clien

Maybe this is related with create-relocatable-package.py in scylla-python3 repo, will take a look.

syuu1228 · 2023-04-03T09:45:05Z

Probably because we haven't added mtime of TarInfo object:

    ti = tarfile.TarInfo(name=os.path.join("bin", pybin))
    ti.size = len(thunk)
    ti.mode = 0o755
    ar.reloc_addfile(ti, fileobj=io.BytesIO(thunk))
    
    ti = tarfile.TarInfo(name=os.path.join("bin", "python3"))
    ti.type = tarfile.SYMTYPE
    ti.linkname = pybin
    ar.reloc_addfile(ti)

https://github.com/scylladb/scylla-python3/blob/master/scripts/create-relocatable-package.py

I will send a patch.

We mistakenly haven't added mtime while creating TarInfo object, we need to add it. Fixes scylladb/scylladb#13415

syuu1228 · 2023-04-03T09:57:21Z

Sent a patch at scylladb/scylla-python3#37

denesb · 2023-04-14T08:41:43Z

Looks like "Fixes" work cross-repos too. scylladb/scylla-python3#37 had a Fixes scylladb/scylladb#13415 and that resulted as this issue being closed as Done after I merged it.
Note that the update of tools/python3 submodule was only queued to next yet, it is not yet in master at the time this writing.

juliayakovlev · 2023-09-07T11:10:11Z

@DoronArazii
This issue was reproduced with 2023.1. Looks like this fix is missed there
Should the issue be opened in enterprise?

Issue description

This issue is a regression.
It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 1 nodes (i3.large)

Scylla Nodes used in this run:

artifacts-oel76-jenkins-db-node-80ba2e34-1 (54.195.135.134 | 10.4.2.34) (shards: 2)

OS / Image: ami-0d6a24fe35fdf5dc4 (aws: eu-west-1)

Test: artifacts-oel76-test
Test id: 80ba2e34-2e7e-4a3f-a371-2cd5009ad8e1
Test name: enterprise-2023.1/artifacts-offline-install/artifacts-oel76-test
Test config file(s):

oel76.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 80ba2e34-2e7e-4a3f-a371-2cd5009ad8e1
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 80ba2e34-2e7e-4a3f-a371-2cd5009ad8e1

Logs:

db-cluster-80ba2e34.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/80ba2e34-2e7e-4a3f-a371-2cd5009ad8e1/20230906_100245/db-cluster-80ba2e34.tar.gz
sct-runner-events-80ba2e34.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/80ba2e34-2e7e-4a3f-a371-2cd5009ad8e1/20230906_100245/sct-runner-events-80ba2e34.tar.gz
sct-80ba2e34.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/80ba2e34-2e7e-4a3f-a371-2cd5009ad8e1/20230906_100245/sct-80ba2e34.log.tar.gz

Jenkins job URL
Argus

mykaul · 2023-09-07T13:16:46Z

Most likely - lacked the 'backport ...' labels.

juliayakovlev · 2023-09-07T13:44:52Z

Most likely - lacked the 'backport ...' labels.

@mykaul
The issue is reproduced in 2023.1, enterprise. Do we need to open the issue in enterprise? Or add backport lable to 2023.1?

mykaul · 2023-09-07T13:46:23Z

I've added the labels for backport, I think it should go via this path.

Annamikhlin · 2023-09-13T09:27:19Z

also failed on enterprise: https://jenkins.scylladb.com/job/scylla-enterprise/job/artifacts-offline-install/job/artifacts-oel76-test/122/

@denesb @mykaul - isn't this fix already merged to enterprise?

mykaul · 2023-09-13T10:08:11Z

I think we need an update of https://github.com/scylladb/scylla-python3 submodule update into Scylla (Enterprise) , no?

denesb · 2023-09-13T13:47:19Z

09:46 $ git log --oneline -3
7d49a46 (HEAD, origin/enterprise) dist: Support FIPS mode
30b8fc2 (origin/master, origin/HEAD) create-relocatable-package.py: fix timestamp of executable files
d2f57dd dist: redhat: provide only a single version

The fix is in enterprise already.

mykaul · 2023-09-13T13:49:35Z

So we'll need to understand better why it failed then.

avikivity · 2023-10-31T14:09:22Z

Why is this being considered for backport? the timestamp of some files doesn't strike me as important.

mykaul · 2023-10-31T14:29:20Z

tar: scylla-5.3.0dev/scylla-python3/bin/python3.11: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0dev/scylla-python3/bin/python3: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0dev/scylla-python3/bin/geomet: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0dev/scylla-python3/bin/traceback-with-variables: implausibly old time stamp 1970-01-01 00:00:00
tar: scylla-5.3.0~dev/scylla-python3/bin/scylla-api-client: implausibly old time stamp 1970-01-01 00:00:00

I don't understand if it's a warning or an error. Assuming the latter, then it might be worthy of a backport to 5.2 (2023.1 LTS?)

avikivity · 2023-10-31T16:16:54Z

It's a warning. But if it's user-visible we can backport it.

We mistakenly haven't added mtime while creating TarInfo object, we need to add it. Fixes scylladb/scylladb#13415 Closes: #37 (cherry picked from commit 30b8fc2)

* tools/python3 cf7030a...6ad2e5a (1): > create-relocatable-package.py: fix timestamp of executable files Fixes #13415.

avikivity · 2023-10-31T16:20:25Z

Backported to 5.2. 5.1 deemed unnecessary because few new installations are likely to use it (and installing from tarball is rare).

* tools/python3 cf7030a...6ad2e5a (1): > create-relocatable-package.py: fix timestamp of executable files Fixes #13415.

Annamikhlin · 2023-11-19T08:01:12Z

The issue seen on 2024.1 - https://jenkins.scylladb.com/job/enterprise-2024.1/job/artifacts-offline-install/job/artifacts-oel76-test/2/
and we still have it on enterprise - https://jenkins.scylladb.com/job/scylla-enterprise/job/artifacts-offline-install/job/artifacts-oel76-test/117/

mykaul · 2023-12-13T12:41:13Z

@avikivity - do we need a 5.4 backport for this, based on the comment by @Annamikhlin above?

denesb · 2023-12-13T14:51:43Z

@avikivity - do we need a 5.4 backport for this, based on the comment by @Annamikhlin above?

This is already in 5.4 as scylladb/scylla-python3@30b8fc2

mykaul · 2023-12-13T15:52:21Z

So it's not fixed, which is even more worrying :-/

denesb · 2023-12-14T10:30:25Z

So it's not fixed, which is even more worrying :-/

I mean the fix is already in 5.4, 5.2 and 5.1. No more backporting is needed.

fgelcer added the triage/master Looking for assignee label Apr 2, 2023

syuu1228 added a commit to syuu1228/scylla-python3 that referenced this issue Apr 3, 2023

create-relocatable-package.py: fix timestamp of executable files

ffe09c9

We mistakenly haven't added mtime while creating TarInfo object, we need to add it. Fixes scylladb/scylladb#13415

syuu1228 mentioned this issue Apr 3, 2023

create-relocatable-package.py: fix timestamp of executable files scylladb/scylla-python3#37

Closed

DoronArazii assigned syuu1228 and yaronkaikov Apr 3, 2023

DoronArazii added P2 High Priority and removed triage/master Looking for assignee labels Apr 3, 2023

DoronArazii added this to the 5.3 milestone Apr 3, 2023

denesb closed this as completed in scylladb/scylla-python3@30b8fc2 Apr 14, 2023

mykaul added Backport candidate backport/5.2 Issues that should be backported to 5.2 branch once they'll be fixed Requires-Backport-to-5.1 labels Sep 7, 2023

avikivity added a commit that referenced this issue Oct 31, 2023

Update tools/pythion3 submodule (tar file timestamps)

d904199

* tools/python3 cf7030a...6ad2e5a (1): > create-relocatable-package.py: fix timestamp of executable files Fixes #13415.

avikivity removed the Backport candidate label Oct 31, 2023

tgrabiec pushed a commit that referenced this issue Nov 2, 2023

Update tools/pythion3 submodule (tar file timestamps)

9967c0b

* tools/python3 cf7030a...6ad2e5a (1): > create-relocatable-package.py: fix timestamp of executable files Fixes #13415.

mykaul added Backport candidate and removed Backport candidate backport/5.2 Issues that should be backported to 5.2 branch once they'll be fixed labels Dec 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

artifacts offline install - implausibly old time stamp 1970-01-01 00:00:00 #13415

artifacts offline install - implausibly old time stamp 1970-01-01 00:00:00 #13415

fgelcer commented Apr 2, 2023

Logs:

fgelcer commented Apr 2, 2023

yaronkaikov commented Apr 2, 2023

syuu1228 commented Apr 3, 2023

syuu1228 commented Apr 3, 2023

syuu1228 commented Apr 3, 2023

denesb commented Apr 14, 2023

juliayakovlev commented Sep 7, 2023 •

edited

Logs:

mykaul commented Sep 7, 2023

juliayakovlev commented Sep 7, 2023

mykaul commented Sep 7, 2023

Annamikhlin commented Sep 13, 2023

mykaul commented Sep 13, 2023

denesb commented Sep 13, 2023

mykaul commented Sep 13, 2023

avikivity commented Oct 31, 2023

mykaul commented Oct 31, 2023

avikivity commented Oct 31, 2023

avikivity commented Oct 31, 2023

Annamikhlin commented Nov 19, 2023

mykaul commented Dec 13, 2023

denesb commented Dec 13, 2023

mykaul commented Dec 13, 2023

denesb commented Dec 14, 2023

artifacts offline install - implausibly old time stamp 1970-01-01 00:00:00 #13415

artifacts offline install - implausibly old time stamp 1970-01-01 00:00:00 #13415

Comments

fgelcer commented Apr 2, 2023

Issue description

Impact

How frequently does it reproduce?

Installation details

Logs:

fgelcer commented Apr 2, 2023

yaronkaikov commented Apr 2, 2023

syuu1228 commented Apr 3, 2023

syuu1228 commented Apr 3, 2023

syuu1228 commented Apr 3, 2023

denesb commented Apr 14, 2023

juliayakovlev commented Sep 7, 2023 • edited

Issue description

Impact

How frequently does it reproduce?

Installation details

Logs:

mykaul commented Sep 7, 2023

juliayakovlev commented Sep 7, 2023

mykaul commented Sep 7, 2023

Annamikhlin commented Sep 13, 2023

mykaul commented Sep 13, 2023

denesb commented Sep 13, 2023

mykaul commented Sep 13, 2023

avikivity commented Oct 31, 2023

mykaul commented Oct 31, 2023

avikivity commented Oct 31, 2023

avikivity commented Oct 31, 2023

Annamikhlin commented Nov 19, 2023

mykaul commented Dec 13, 2023

denesb commented Dec 13, 2023

mykaul commented Dec 13, 2023

denesb commented Dec 14, 2023

juliayakovlev commented Sep 7, 2023 •

edited