problem with mupif file transfer

[grauchs]

When trying to transfer files (which worked without problems in the composelector-mupif version) with the commands:

for inpFile in inpFiles:
       try:
              pf = self.abaqusJobMan.getPyroFile(macroSolver.getJobID(), inpFile, 'wb', buffSize = 1024*1024)
       except Exception as err:
              print(' ***    getPyroFile ' + repr(err))
       try:
              pyroutil.uploadPyroFile(os.path.join(inpDir,inpFile), pf, hkey, size = 1024*1024, compressFlag = True)
       except Exception as err:
              print(' ***    uploadPyroFile ' + repr(err))

I get the following error messages:

***    getPyroFile TypeError('__init__() takes 1 positional argument but 4 were given')
***    uploadPyroFile AttributeError("module 'mupif.pyroutil' has no attribute 'uploadPyroFile'")

I noticed there is a copy method in the pyrofile module, but I don’t see what arguments have to be given. Does this replace the uploadpyrofile method? What about the getpyrofile error?

[stanislavsulc]

The uploadPyroFile function is no longer included in the pyroutil module (or anywhere else).

The mentioned mupif.PyroFile.copy function is used to create a local/remote copy of a local/remote file. Both arguments of the function (src and dst) can be a string path (local file), a PyroFile instance or its remote representation as Pyro5.api.Proxy. Example of creating a local copy of a remote file is: mupif.PyroFile.copy(remote_file, '/path/to/save/the/file.txt')
Use of this function can be found in examples in file https://github.com/mupif/mupif/blob/master/examples/models.py

Was this answer helpful?

[grauchs]

I manage to copy the files now, after creating an intance of a pyro file and using the pyrofile.copy method. As the method getpyrofile of the simplejobmanager and jobmanager module currently doesn't work - and I suppose the uploadfile method doesn't work either - does this mean that both methods will be abandonned and replaced by pyrofile.copy?

[grauchs]

Now there seems to be a different problem: the copy of the file is not complete, and strangely enough has the length of 1048576, which is the buffer size internally defined in the pyrofile module. Is there something wrong with the copy method? The file size remains the same even if I give an alternative buffer size while creating the pyrofile instance.

[bpatzak]

Can you provide a snapshot of the code & sample file to document the issue?

[grauchs]

Here is the code:

# copy Abaqus input files to server work directories
for inpFile in inpFiles:
     try:
          pf = mp.PyroFile(filename = os.path.join(workdir,inpFile), mode = 'wb', buffsize=1024*1024*4)
     except Exception as err:
          print(' ***    getPyroFile ' + repr(err))
     try:
          mp.PyroFile.copy(os.path.join(inpDir, inpFile),pf)
          print('COPIED ABAQUS', inpFile)

The file I use has 3 MB. It is the file composelector_panel-quasi-v5.inp from the composelector project. Please note that a short file of less than 300 bytes got copied correctly

[bpatzak]

I acknowledge the problem. This is for my group:
procedure to replicate problem:

1. generate random file

   $ dd if=/dev/urandom of=in.bin count=4096 bs=1024
   

2. test script:

   import mupif as mp
   import os
   pf = mp.PyroFile(filename = "out.bin", mode = 'wb')
   mp.PyroFile.copy("in.bin", pf)
   print('FILE COPIED')

3. compare in.bin and out.bin

[grauchs]

now there seems to be a different problem:
when using the method pyrofile to create an instance of a file to be copied subsequently, I run into permission issues. When the server has been started under my username, I manage to create the instance and copy, whereas is the server was started by a different user (member of the same group), I get a permission denied. In fact, the local working directory of the job on the server is created by the jobmanager with ownership 755. This was never an issue in mupif/composelector. One noteable difference is that with mupif/Deema, the pyrofile is created as a pyrofile instance in the workflow run by the user, whereas before getpyrofile (used for creating the file) was a method of the jobmanager, so probably having the same access right as the jobmanager. Could this be the cause of the problem?