Ruciax tsm fixes #119
Ruciax tsm fixes #119
Conversation
…, changed massive-ruciax logic
cax/main.py
Outdated
| @@ -830,7 +826,7 @@ def massiveruciax(): | |||
| dd = divmod(diff.total_seconds(), 60) | |||
|
|
|||
| #delete script: | |||
| qsub.delete_script( sc ) | |||
| #qsub.delete_script( sc ) | |||
There was a problem hiding this comment.
Do you really want to keep the script?
There was a problem hiding this comment.
of course not! Thank you checking.
cax/tasks/data_mover.py
Outdated
| @@ -592,10 +592,20 @@ def copy_tsm(self, datum, destination, method, option_type): | |||
| for name in dirs: | |||
| list_folders.append(name) | |||
|
|
|||
| #Sanity check from zero sized files during the upload | |||
| # find *.pickles or *log files which are size of zere and delte them before upload | |||
There was a problem hiding this comment.
This comment says you check if file is *.pickles or *log, but I don't see this check below. For example, a zero-sized .zip could indicate a problem, but would just get deleted here.
There was a problem hiding this comment.
This is true. I put this in the comment section to make the purpose clear (*pickle and eventually an empty log file).
For the deletion process later, I would say that a zero sized zip file indicates a problem based on pax. But we also should not upload these files then. For rucio it does not make a difference and it will complain about any zero sized file then.
BTW: I have never seen a zero sized *zip file ever.
cax/tasks/data_mover.py
Outdated
| @@ -613,8 +623,8 @@ def copy_tsm(self, datum, destination, method, option_type): | |||
| else: | |||
| logging.info("Pre-test of %s counts %s files for tape upload [succcessful]", raw_data_path+raw_data_filename, len(list_files)) | |||
|
|
|||
|
|
|||
| #Do a checksum pre-test: | |||
| exit() | |||
There was a problem hiding this comment.
Do you mean to keep this here?
|
Try to also fix the codacy style issues (click "Details" below or run |
|
I applied many lint/flake8 fixes to #114, so you might want to merge that first, then sync with this branch. Then you can check what changes need to be made with Also, does your python really work with only 2-space indents? |
|
@pdeperio If we can avoid creating zero sized acquisition monitor files in pax we need to think if we still want to keep my "protection" of uploading zero sized files to rucio/tape. For example there could also be a zero sized log file (I have seen this in the past). About the style issues: I will fix this latest before the final merging, thanks for reminding. And yes, "my python" works also with 2-space intends. But I think this is somehow pre-definied in Kate (Linux editor of the KDE environment) or the notebooks which I run on midway. I try to be more careful about it in the future. |
|
@pdeperio I made some code review with pylint and flake8 now for: Should be fine now |
…, changed massive-ruciax logic
…o ruciax_tsm_fixes
Dear all,
I finished a fairly huge bug fix and maintenance campaign in ruciax and tape upload related tools. This contains the following changes:
4.1 Raw data sets which are marked by "error" are skipped
4.2 Re-upload raw data (status=RSEreupload) automatically with massive-ruciax
4.3 Uploads are separated from verification now.
6.Both
tools (ruciax and cax-tsm) delete zero sized files in the raw data folder on xe1t-datamanager before the upload to tape or rucio is started. This becomes necessary to not be inconsistence with data on tape and rucio (In particular if we would like to restore them later). There are two consequences:6.1.
Incremental upload of old raw data to tape (in case something has changed on xe1t-datamanager and the raw data set was already uploaded to tape) is not possible anymore. We need to delete the tape backup and re-upload it again. Please not that this will not happen so often, or let's almost never.6.2.
We still need remove zero sized files from the rucio catalog. This will happen soon.6.3. Latest pax version avoids to produce zero-sized files. No need to change anything here anymore.
6.4. Implication: Replace zero-sized raw data in the rucio catalog by one which has a size (additional script)
This request comes from Benedikt to distribute our raw data transfers on the "European" site to several locations. Therefore the the destination_rse and destination_lifetime key of the "rucio-rule" takes now arguments like this:
"destination_rse": ["LNGS_USERDISK", "random:CCIN2P3_USERDISK,CNAF_USERDISK,NIKHEF_USERDISK,WEIZMANN_USERDISK", "UC_OSG_USERDISK"], "destination_livetime":{ "LNGS_USERDISK": "432000", "random:CCIN2P3_USERDISK,CNAF_USERDISK,NIKHEF_USERDISK,WEIZMANN_USERDISK": "-1", "UC_OSG_USERDISK": "-1" },This example draw one RSE from the list (random:...) which is given and use this to create the transfer destinations.
I tested the code on xe1t-datamanager already