Create dir2ram

[Reed97123]

Hi,

Following on your good work, I've created a new version of folder2ram called dir2ram which extends the work to accomplish most of your todo list as well as fixing some of the bugs that I've experienced (long directory mangling, services needing to be stopped).

I was hoping you could give feedback on features and any bugs you encounter.

Summary:

• Stops dependent service(s), starts the tmpfs|ramfs drive, then restarts services for mounts. Same for unmounts as well.
• Allows tmpfs or ramfs RAM drives.
• Allows you to set a maximum size of drive.
• Checks the size of the directory and warns if it is larger than specified in the config file.
• Optionally prunes logs back to user specified level before syncing to disk.
• Automated daily syncs back to disk.
• Lots of sanity and safety checks.

Example Config file:

EXAMPLE:

Name FS MB LogHist Dependent_Service(s) Dir_To_Cache

smart tmpfs 8 5 smartd,smartmontools /var/lib/smartmontools

sickbeard tmpfs 256 5 sickbeard /var/opt/sickbeard

couch tmpfs 256 5 couchpotato /home/couchpotato

samba tmpfs 16 5 samba /var/cache/samba
ntp tmpfs 8 5 ntp /var/lib/ntp

minidlna tmpfs 16 5 minidlna /var/cache/minidlna

Covered by folder2ram:

log tmpfs 256 8 /var/log
vartmp tmpfs 512 5 /var/tmp
spool tmpfs 32 5 postfix /var/spool
rrd tmpfs 8 5 rrdcached /var/lib/openmediavault/rrd
rrdcached tmpfs 256 5 rrdcached /var/lib/rrdcached
monit tmpfs 8 5 monit /var/lib/monit
php tmpfs 8 5 php5-fpm /var/lib/php5

Example Status:
Name Space FileS Real Directory Mounted tmpfs|ramfs
==== ===== ===== ============== ===================
rrdcached 16M/256M tmpfs /var/dir2ram/dir2ram_rrdcached /var/lib/rrdcached
ntp 1M/8M tmpfs /var/dir2ram/dir2ram_ntp /var/lib/ntp
vartmp 0M/512M tmpfs /var/dir2ram/dir2ram_vartmp /var/tmp
rrd 1M/8M tmpfs /var/dir2ram/dir2ram_rrd /var/lib/openmediavault/rrd
php 1M/8M tmpfs /var/dir2ram/dir2ram_php /var/lib/php5
smart 0M/8M tmpfs /var/dir2ram/dir2ram_smart /var/lib/smartmontools
log 21M/256M tmpfs /var/dir2ram/dir2ram_log /var/log
monit 1M/8M tmpfs /var/dir2ram/dir2ram_monit /var/lib/monit
spool 2M/32M tmpfs /var/dir2ram/dir2ram_spool /var/spool
samba 1M/16M tmpfs /var/dir2ram/dir2ram_samba /var/cache/samba

[bobafetthotmail]

oh just browsed fast through it, and it looks cool.
you rewrote most parts and it's significantly more pro now.

This weekend will have a more through look into it.

[bobafetthotmail]

gave it a more through look. Is there a compelling reason for it to be written in perl?

I'm not seeing anything that can't be done with shell scripts too, and I'd rather use shell scripts myself.

[Reed97123]

Hi,

For me shell scripts offer the maximum portability and comparability
because it is ensured people can run your shell script without anything
else being installed. But their limitation is that they are very limited
in their features and what you can do with them.

Programming languages on the other hand are very flexible, but often take
compiling or have other installation requirements.

Perl, as a scripting language is a good mix of both benefits. It has near
universal support on all systems and is very feature packed.

Off the top of my head I used argument processing, hashes, functions,
formatted printing (printf), various regular expression processing, various
array functions. I wouldn't have known where to start trying to implement
this in a shell script. I'm fairly certain shell scripts don't have an
equivalent to perl hash data structures. I also heavily relied upon Perl's
regular expression functions which I wouldn't know how or if it is possible
to do in a shell script.

I read your GitHub 'Todo List' and I was able to implement nearly
everything on your list.

I was curious if there was any other features that should be added from
your perspective? I'd be happy to add it in now.

Also, in the time between now and when I posted the version I made a few
bug fixes and minor changes. Should I upload the new version to GitHub?

-Reed

On Fri, Jan 22, 2016 at 3:38 AM, bobafetthotmail notifications@github.com
wrote:

gave it a more through look. Is there a compelling reason for it to be
written in perl?

I'm not seeing anything that can't be done with shell scripts too, and I'd
rather use shell scripts myself.

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[bobafetthotmail]

With all due respect because you clearly are a better programmer than me...

I know shell (sh) is limited and I was already thinking about migrating, but I was looking mostly towards bash (which is similar to sh and I understand it decently, it is also the linux default CLI afaik).

Bash can do the things you mentioned as perl advantages.

• argument processing http://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash
• functions :::: I can assure you that there are plenty of functions in my shell script, bash does the same. Otherwise I did not understand this point.
• hash tables :::: bash has them http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash
• formatted printing (printf) ::: there is a "printf" component of coreutils to print formatted text from whatever script language, write a "man printf" from terminal to see what it can do. I'm suspecting it is the same thing called by perl.
• regular expression processing ::: that's done by calling awk and sed text processors in my own script already, to read the config file for example, or to turn paths into safe names for folders. It's probably easier with perl though, they are VERY through, awk has huge manuals online.
• arrays :::: yes in bash, also with strings http://tldp.org/LDP/abs/html/arrays.html

"I was curious if there was any other features that should be added from your perspective? I'd be happy to add it in now."

The ability to read/parse multiple config files. like most programs do, (like samba) with a /etc/folder2ram/config.d where the lines from the second file override the (same) lines from the first one.

The ability to mount folders to disk, with the synching too. So you can move particular folders inside the RAID volume or data drives/whatever spinning drives.
Which will be useful for programs/caches that cannot fit in ram but can't stay on flash either.

Also an option for ram drives to get a frequent synching of ram drives into data drives (either alone or in addition to synching to system drive), like for logs and stuff like that, you can have an option that synches to data drives every X minutes if there is enough write activity in the last X minutes to justify their use, otherwise don't sync and let the data drives go to sleep.

I also wanted to experiment with a script that makes a squashfs of the folder, with a script run at boot that loads and keeps the squashfs in the ram.
When it is time to update/upgrade it unmounts the squashfs, runs the update, makes again the squashfs.
More for experimenting and seeing if there is any performance increase, this isn't terribly important though.

" Also, in the time between now and when I posted the version I made a few bug fixes and minor changes. Should I upload the new version to GitHub?"

I'm not going to accept Perl scripts in my repo, as I never really used it (yes I'm not kidding, don't look at me like that), and I don't have the time to learn how to use it decently, so I won't be able to maintain what you send.

If you prefer keeping it in Perl, open your own repository, upload it there, maybe mention me in the credits or something, and become the lead developer. You already have a new name for your script lol.
I could help with "debianizing" (adding the stuff needed for packaging it for Debian) and changing the OMV plugin to use dir2ram (if you need my help for these at all).

[Reed97123]

Hi,

I didn't mean to put you on the defensive to defend Bash. I actually don't
know much about programming in Bash. I knew enough to follow the logic of
your script and understand what you were doing. But it would have been
beyond my expertise to add in all the 'todo list' items using Bash. I've
used Perl on and off for 16 years, so I was simply more comfortable using
it. I already knew how to do 100% of what was needed in Perl whereas I'd
probably only know how to do 20% of it in Bash and I'd have to spend a very
long time learning.

I did find a very interesting comparison of the two when you mentioned
Bash's capabilities. It was written by someone who knows both very well:
http://blogs.perl.org/users/buddy_burden/2012/04/perl-vs-shell-scripts.html

I would agree that I should create my own repo. I didn't want to take over
your project, but as I'm not good with Bash, I couldn't figure any other
way of contributing and accomplishing the needed coding.

Items you mentioned:

• Multiple configuration files. So this would allow one configuration file
  from the developer and a second file for the user so they can each change
  different files and have both loaded so they don't conflict with each
  others changes?

For example:

• /etc/dir2ram/dir2ram.conf (Developer controlled file)
• /etc/dir2ram/dir2ram_user.conf (User controlled file)
  That was an example off the top of my head. Let me know if you have a
  better idea. How should it handle duplicates? Currently I ignore
  redefinitions/duplicates that occur after the first. But I could easily
  make it so that duplicates overwrite the previous definition. I guess it
  just depends on what we think would be most useful.
• Moving directories from one disk to another. So for example you have an
  OS directory on a flash drive and it is too big for memory, so you want it
  to effectively be moved to a spin drive/SSD so it doesn't wear out the USB
  flash drive?
• Squashfs. I did some reading and I'm curious as to your expected use
  cases. Squashfs is typically used for live CD Linux distributions and
  other embedded applications where the read-only aspect isn't a problem. If
  your desire is to have a compressed directory in memory, perhaps ZRam is a
  better solution. It would allow us to turn on compression for specified
  directories. Perhaps similar to this web site:
  http://askubuntu.com/questions/130374/ramdisk-compressed-writeable-no-swap
• One idea I had was to allow users to specify directories directly from
  the OMV GUI interface. This would be similar to Fail2ban's GUI interface.
  They could enter directories and view a nice summary of what was done there
  without needing to modify or look at the actual conf file. This might
  decrease the need for multiple configuration files (earlier todo list item).

I just created a GitHub repository, with the latest version:
https://github.com/Reed97123/dir2ram

I think before considering using it for the flashmemory plugin I'd like to
do more testing on it and add in any of the features we think are desired.
The config file addition would be super-simple, but might not be needed if
the GUI interface is added. I'm also pretty excited about compressed
directories in memory, which I think would be a lot of fun to add.

As this is my first GitHub project, any help and advice is definitely
welcome!

-Reed

On Sat, Jan 23, 2016 at 12:12 AM, bobafetthotmail notifications@github.com
wrote:

With all due respect because you clearly are a better programmer than me...

I know shell (sh) is limited and I was already thinking about migrating,
but I was looking mostly towards bash (which is similar to sh and I
understand it decently, it is also the linux default CLI afaik).

Bash can do the things you mentioned as perl advantages.

-argument processing
http://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash

-functions :::: I can assure you that there are plenty of functions in my
shell script already, go figure in bash. Otherwise I did not understand
this point.

-hash tables :::: bash has them
http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash

-formatted printing (printf) ::: there is a "printf" component of
coreutils to print formatted text from whatever script language, write a
"man printf" from terminal to see what it can do. I'm suspecting it is the
same thing called by perl.

• regular expression processing ::: that's done by calling awk and sed
  text processors in my own script already, to read the config file for
  example, or to turn paths into safe names for folders. It's probably easier
  with perl though, they are VERY through, awk has huge manuals online.

-arrays :::: yes in bash, also with strings
http://tldp.org/LDP/abs/html/arrays.html

"I was curious if there was any other features that should be added from
your perspective? I'd be happy to add it in now."

The ability to read/parse multiple config files. like most programs do,
(like samba) with a /etc/folder2ram/config.d where the lines from the
second file override the (same) lines from the first one.

The ability to mount folders to disk, with the synching too. So you can
move particular folders inside the RAID volume or data drives/whatever
spinning drives.
Which will be useful for programs/caches that cannot fit in ram but can't
stay on flash either.

I also wanted to experiment with a script that makes a squashfs of the
folder, with a script run at boot that loads and keeps the squashfs in the
ram.
When it is time to update/upgrade it unmounts the squashfs, runs the
update, makes again the squashfs.
More for experimenting and seeing if there is any performance increase,
this isn't terribly important though.

" Also, in the time between now and when I posted the version I made a few
bug fixes and minor changes. Should I upload the new version to GitHub?"

I'm not going to accept Perl scripts in my repo, as I never really used it
(yes I'm not kidding, don't look at me like that), and I don't have the
time to learn how to use it decently, so I won't be able to maintain what
you send.

If you prefer keeping it in Perl, open your own repository, upload it
there, and maybe mention me in the credits or something, and become the
lead developer. You already have a new name for your script lol.
I could help with "debianizing" (adding the stuff needed for packaging it
for Debian) and changing the OMV plugin to use dir2ram (if you need my help
for these at all).

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[Reed97123]

Hi,

Update about zRam. It is possible to use if you load the backported kernel
3.16. The default OMV Wheezy kernel (3.2) doesn't yet support zRam.

It is actually a great solution, providing a mechanism to have read/write
access to a compressed RAM directory. It is a little tedious to add new
drives after the initial setup, all drives must be removed and module
removed and then readded with the new additional drives. They have added a
hot-add features, but it's not in even the backported kernel. Even given
the limitation, I think it would make a fantastic addition to the program.

-Reed

On Sat, Jan 23, 2016 at 10:56 AM, Reed Nelson reed.public@gmail.com wrote:

Hi,

I didn't mean to put you on the defensive to defend Bash. I actually
don't know much about programming in Bash. I knew enough to follow the
logic of your script and understand what you were doing. But it would have
been beyond my expertise to add in all the 'todo list' items using Bash.
I've used Perl on and off for 16 years, so I was simply more comfortable
using it. I already knew how to do 100% of what was needed in Perl whereas
I'd probably only know how to do 20% of it in Bash and I'd have to spend a
very long time learning.

I did find a very interesting comparison of the two when you mentioned
Bash's capabilities. It was written by someone who knows both very well:
http://blogs.perl.org/users/buddy_burden/2012/04/perl-vs-shell-scripts.html

I would agree that I should create my own repo. I didn't want to take
over your project, but as I'm not good with Bash, I couldn't figure any
other way of contributing and accomplishing the needed coding.

Items you mentioned:

• Multiple configuration files. So this would allow one configuration
  file from the developer and a second file for the user so they can each
  change different files and have both loaded so they don't conflict with
  each others changes?

For example:

• /etc/dir2ram/dir2ram.conf (Developer controlled file)
• /etc/dir2ram/dir2ram_user.conf (User controlled file)
  That was an example off the top of my head. Let me know if you have a
  better idea. How should it handle duplicates? Currently I ignore
  redefinitions/duplicates that occur after the first. But I could easily
  make it so that duplicates overwrite the previous definition. I guess it
  just depends on what we think would be most useful.
• Moving directories from one disk to another. So for example you have an
  OS directory on a flash drive and it is too big for memory, so you want it
  to effectively be moved to a spin drive/SSD so it doesn't wear out the USB
  flash drive?
• Squashfs. I did some reading and I'm curious as to your expected use
  cases. Squashfs is typically used for live CD Linux distributions and
  other embedded applications where the read-only aspect isn't a problem. If
  your desire is to have a compressed directory in memory, perhaps ZRam is a
  better solution. It would allow us to turn on compression for specified
  directories. Perhaps similar to this web site:
  http://askubuntu.com/questions/130374/ramdisk-compressed-writeable-no-swap
• One idea I had was to allow users to specify directories directly from
  the OMV GUI interface. This would be similar to Fail2ban's GUI interface.
  They could enter directories and view a nice summary of what was done there
  without needing to modify or look at the actual conf file. This might
  decrease the need for multiple configuration files (earlier todo list item).

I just created a GitHub repository, with the latest version:
https://github.com/Reed97123/dir2ram

I think before considering using it for the flashmemory plugin I'd like to
do more testing on it and add in any of the features we think are desired.
The config file addition would be super-simple, but might not be needed if
the GUI interface is added. I'm also pretty excited about compressed
directories in memory, which I think would be a lot of fun to add.

As this is my first GitHub project, any help and advice is definitely
welcome!

-Reed

On Sat, Jan 23, 2016 at 12:12 AM, bobafetthotmail <
notifications@github.com> wrote:

With all due respect because you clearly are a better programmer than
me...

I know shell (sh) is limited and I was already thinking about migrating,
but I was looking mostly towards bash (which is similar to sh and I
understand it decently, it is also the linux default CLI afaik).

Bash can do the things you mentioned as perl advantages.

-argument processing
http://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash

-functions :::: I can assure you that there are plenty of functions in my
shell script already, go figure in bash. Otherwise I did not understand
this point.

-hash tables :::: bash has them
http://stackoverflow.com/questions/1494178/how-to-define-hash-tables-in-bash

-formatted printing (printf) ::: there is a "printf" component of
coreutils to print formatted text from whatever script language, write a
"man printf" from terminal to see what it can do. I'm suspecting it is the
same thing called by perl.

• regular expression processing ::: that's done by calling awk and
  sed text processors in my own script already, to read the config file for
  example, or to turn paths into safe names for folders. It's probably easier
  with perl though, they are VERY through, awk has huge manuals online.

-arrays :::: yes in bash, also with strings
http://tldp.org/LDP/abs/html/arrays.html

"I was curious if there was any other features that should be added from
your perspective? I'd be happy to add it in now."

The ability to read/parse multiple config files. like most programs do,
(like samba) with a /etc/folder2ram/config.d where the lines from the
second file override the (same) lines from the first one.

The ability to mount folders to disk, with the synching too. So you can
move particular folders inside the RAID volume or data drives/whatever
spinning drives.
Which will be useful for programs/caches that cannot fit in ram but can't
stay on flash either.

I also wanted to experiment with a script that makes a squashfs of the
folder, with a script run at boot that loads and keeps the squashfs in the
ram.
When it is time to update/upgrade it unmounts the squashfs, runs the
update, makes again the squashfs.
More for experimenting and seeing if there is any performance increase,
this isn't terribly important though.

" Also, in the time between now and when I posted the version I made a
few bug fixes and minor changes. Should I upload the new version to GitHub?"

I'm not going to accept Perl scripts in my repo, as I never really used
it (yes I'm not kidding, don't look at me like that), and I don't have the
time to learn how to use it decently, so I won't be able to maintain what
you send.

If you prefer keeping it in Perl, open your own repository, upload it
there, and maybe mention me in the credits or something, and become the
lead developer. You already have a new name for your script lol.
I could help with "debianizing" (adding the stuff needed for packaging it
for Debian) and changing the OMV plugin to use dir2ram (if you need my help
for these at all).

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[bobafetthotmail]

Closing this without merging for the reasons already discussed.

I've answered (thoroughly) to your post in an issue I opened in your own new repo Reed97123/dir2ram#1