Out of memory while writing on a LUN

[MMartinezV]

Hi,

I'm testing iscsi-osx on with El Capitan and a ReadyNAS RNDP4420 with ReadyNAS OS 6.4.1 and I'm experimenting an strange issue.

I have installed it using the released (beta) dmg. It seems to work nice. I'm able to add the target, login and then format the disk, but when I write files on the Volume the used wired memory starts to grow until the system stops working.

If I write a 1GB file on the LUN (with: dd if=/dev/zero of=test.img bs=1024k count=1000) the wired mem used grows 1GB. It seems that the system doesn't free the buffers (or similar).

Looking at /Library/Logs/DiagnosticReports I see the following crash report.

Can you help me?

Thanks in advance,

Manuel Martínez

iscsictl_2016-01-25-143048_vm-elcapitan-server.crash.txt

[MMartinezV]

Update:

I've repeated the install on a new ElCapitan (virtualized on KVM) and it also fails.

In fact, I've seen that the installer in the release image file (the .dmg package) does not work (out of the box) on my systems, because the directories /usr/local/bin/ and /Library/PrivilegedHelperTools/ doesn't exist (at least on my recently installed systems) and they need to be created.

I have monitored a little more what's happening and using the command 'zprint -t' I can see that the kernel module 'com.github.iscsi-osx.iSCSIInitiator' uses more and more memory as I do writes on the LUN.

Hope this helps.

Regards,

Manuel

P.D.: We've been able to use without any problems (without memory leaks) the iscsi initiator on another El Capitan Machine. It's a MBP of a developer with many developer tools and utilities on it (could be a dependency on a library??). Tomorrow I'll do some more tests

[nsinenian]

Thanks for testing. I believe the memory leak had to do with an incorrect parameter to an IOFree() call. It should now be fixed (commit 2a3bc44 on the develop branch). You'll need to manually build and install per instructions in the Wiki.

Please let me know if the issue is resolved. As for the issue with the installer, it doesn't create the folder (PrivilegedHelperTools) if it doesn't exist so that installation script is likely throwing an error there. I'll fix that script but it doesn't make sense to release that until the leak issue has been resolved.

[MMartinezV]

Hi,

I've build and install the developer branch. And it is still failling. (Now, with the dev branch I see a lot of info in the /var/log/iscsid file, and in fact the transfer is very slow compared with the release version).

If I transfer 1Gbyte (doing a dd if=/dev/zero of=test.img bs=1024k count=1000) and then I do a 'zprint -t', I can see that the module com.github.iscsi-osx.iSCSI is using 1GByte memory, so I think that IOFree() is not working at all (it's exactly the same amount of mem).

More info. If I unload the iSCSIInitiator.kext, the wired memory does not get freed, then I see something like ( site 0xffffff80127f70d0 ....).

Thanks,

Manuel

[MMartinezV]

To do a test, I've commented the line IOFree(data,connection->maxSendDataSegmentLength) in the ProcessDataOutForTask() method, rebuilded and installed again.

It does the same. The mem is allocated but not freed.

Regards,

Manuel

[nsinenian]

So the fix wasn't a hunch, I actually saw a difference on my system (the wired memory and the ioalloccount didn't grow indefinitely as before). Did you completely remove the kext (e.g., uninstall.sh, reboot, and rebuild/reinstall? I want to make sure that OS X wasn't caching the old one somehow.

[MMartinezV]

I think you are rigth, your patch is working. Without the IOFree() the mem is not freed and grows the allocated mem, but with IOFree() seems to work nice.

I've placed a trace (DLog("iscsi: MANU testings. IOFree activated (or desactivated) \n";) just before the IOFree() call to be sure that I'm using the modified module.

I'll do some more tests but looks good!

Thanks lots!

Manuel

[nsinenian]

Excellent. What still concerns me is that crash log that you had posted. I don't think it is related to the memory issue, as it shows that the command-line tool iscsictl crashed (so after you typed in a command). It may even be crashing after it executes your command, so from your perspective you wouldn't notice any problem unless you looked at the logs. Please let me know if you find additional crash logs after using iscsictl, and I will try to reproduce the issue.

[MMartinezV]

Hi,

I've done many file transfers of 10GB and 20GB and it's working fine.

I've checked again the release version on the developer's MBP that I thought was working but it fails. Sorry for the mistake, it's connected via a slow network and the memory leak also grows slowly... Sorry again.

Yesterday I had another issue. During a file transfer the LUNs got disconnected and reconnected again. The 'dd' command froze and I wasn't able to kill it. I had to reboot.

I've changed the Energy Saver preferences to avoid the posibility of having the disks in 'sleep' state when they are idle. After that I've done many transfers and for now I have not seen any disconnection nor any problem.

I think you can close this issue, and if I have more 'disconnect' issues I'll open a new one.

Regards,

Manuel

[nsinenian]

Thanks for your contribution. I'll wait a bit to see if any other issues
show up and then release a second beta.

If the machine goes to sleep, the initiator will disconnect all sessions
and reconnect them upon wake up. It is difficult to keep connections alive
in this situation since some targets will send pings that the initiator
needs to respond to.

On Thursday, January 28, 2016, MMartinezV notifications@github.com wrote:

Hi,

I've done many file transfers of 10GB and 20GB and it's working fine.

I've checked again the release version on the developer's MBP that I
thought was working but it fails. Sorry for the mistake, it's connected via
a slow network and the memory leak also grows slowly... Sorry again.

Yesterday I had another issue. During a file transfer the LUNs got
disconnected and reconnected again. The 'dd' command froze and I wasn't
able to kill it. I had to reboot.

I've changed the Energy Saver preferences to avoid the posibility of
having the disks in 'sleep' state when they are idle. After that I've done
many transfers and for now I have not seen any disconnection nor any
problem.

I think you can close this issue, and if I have more 'disconnect' issues
I'll open a new one.

Regards,

Manuel

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

[MMartinezV]

Well, for me, the strange thing was that the machine went to sleep while doing a transfer.

This strange behaviour is in the system, not in your sofware...

Regards,

Manuel