S3 endpoint parameter missing??

[gangamaha]

This is more of a question than issue. Am trying to use this tool for our graphite data backups to S3 and wondering where the endpoint is picked up or specified. We have our service endpoints where we store the buckets, but I don't see where it's being passed to tool or defaulted anywhere in script. Kindly advise

[gangamaha]

Is this where the endpoint is being replaced when passed https://github.com/jjneely/whisper-backup/blob/master/whisperbackup/whisperbackup.py#L83 ?

[jjneely]

I'm just passing a AWS Region to boto and letting boto "know" the endpoints for that region and connect. The above is how you would specify a different region.

Its passed through here:

https://github.com/jjneely/whisper-backup/blob/master/whisperbackup/s3.py#L31

If you are trying to use an S3 compatible service, probably some patches will be needed to be able to specify those endpoints.

[gangamaha]

Right, but I think it would be more flexible if tool support passing optional endpoint parameters. we are trying to use it for IBM COS storage(similar to S3) and it would be very beneficial to pass endpoints for our scenarios instead of adding our regions->endpoint mappint in boto endpoint.json file. I can may be work on submitting a patch if you like. Don't think it would be hard. Please advise?

[gangamaha]

I have tested in my local machine by adding --endpoint option to support our needs and it works. Tail of my dev log shows
2018-08-06-11:40:38;INFO;Progress: 925/950 or 97.368421%
2018-08-06-11:40:38;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-LISTEN ...
2018-08-06-11:40:38;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-LAST_ACK ...
2018-08-06-11:40:38;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-CLOSING ...
2018-08-06-11:40:38;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-SYN_RECV ...
2018-08-06-11:40:38;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-CLOSED ...
2018-08-06-11:40:38;INFO;Progress: 930/950 or 97.894737%
2018-08-06-11:40:38;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-ESTABLISHED ...
2018-08-06-11:40:39;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-FIN_WAIT1 ...
2018-08-06-11:40:39;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-SYN_SENT ...
2018-08-06-11:40:39;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-TIME_WAIT ...
2018-08-06-11:40:39;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-CLOSE_WAIT ...
2018-08-06-11:40:39;INFO;Progress: 935/950 or 98.421053%
2018-08-06-11:40:39;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-22-local.tcp_connections-FIN_WAIT2 ...
2018-08-06-11:40:39;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.pickle.active ...
2018-08-06-11:40:40;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.pickle.errors ...
2018-08-06-11:40:40;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.pickle.metricsReceived ...
2018-08-06-11:40:40;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.df-dev-shm.df_complex-reserved ...
2018-08-06-11:40:40;INFO;Progress: 940/950 or 98.947368%
2018-08-06-11:40:40;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.df-dev-shm.df_complex-free ...
2018-08-06-11:40:40;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.df-dev-shm.df_complex-used ...
2018-08-06-11:40:41;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.df-sys-fs-cgroup.df_complex-reserved ...
2018-08-06-11:40:41;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.df-sys-fs-cgroup.df_complex-free ...
2018-08-06-11:40:41;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.df-sys-fs-cgroup.df_complex-used ...
2018-08-06-11:40:41;INFO;Progress: 945/950 or 99.473684%
2018-08-06-11:40:41;INFO;Backup complete
2018-08-06-11:40:41;INFO;Beginning purge operation.
2018-08-06-11:40:41;INFO;Searching remote file store...

And bucket has wsp files
root@ossgmsdataadc0101b:~# ./sls3 --dev ls ossgmsdataadc0101b/local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101b.interface-eth0.if_dropped.rx/
2018-08-06 11:46:53 40 2018-08-06T16:39:22+00:00.sha1
2018-08-06 11:46:53 19789 2018-08-06T16:39:22+00:00.wsp.gz

Will be happy to create a PR once the author gives a go-signal. Please advise, thanks!!

[jjneely]

Test your restores!

But yes, PRs welcome.

[gangamaha]

yes restore works too with endpoint. Tried creating PR but it said permission denied. Is there some restriction to push commits to repo?

[gangamaha]

Created PR #20

Logs for backup

2018-08-06-11:37:14;INFO;Scanning filesystem...
2018-08-06-11:37:14;INFO;Starting backup of 950 whisper files
2018-08-06-11:37:14;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.collectd-write_queue.queue_length ...
2018-08-06-11:37:14;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.df-root.df_complex-used ...
2018-08-06-11:37:14;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.df-root.df_complex-free ...
2018-08-06-11:37:14;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.df-root.df_complex-reserved ...
2018-08-06-11:38:25;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.collectd-write_queue.derive-dropped ...
2018-08-06-11:38:25;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.tcpconns-8888-local.tcp_connections-LISTEN ...
2018-08-06-11:38:26;INFO;Backup: Processing local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.tcpconns-8888-local.tcp_connections-LAST_ACK ...

Some logs after future backups
2018-08-06-12:28:01;INFO;Restoring local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.tcpconns-2003-local.tcp_connections-FIN_WAIT2 from timestamp 2018-08-06T17:25:04+00:00
2018-08-06-12:28:01;INFO;Restoring local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.cpu-0.cpu-steal from timestamp 2018-08-06T17:24:50+00:00
2018-08-06-12:28:01;INFO;Restoring local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101a.irq.irq-MCP from timestamp 2018-08-06T17:24:24+00:00
2018-08-06-12:28:01;INFO;Restoring local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsgwadc0101a.vmem.vmpage_io-memory.in from timestamp 2018-08-06T17:23:12+00:00
2018-08-06-12:28:01;INFO;Restoring local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.processes.ps_state-blocked from timestamp 2018-08-06T17:24:47+00:00
2018-08-06-12:28:01;INFO;Restoring local-dev-vergil.monitoring.US.rg.adc01.servers.ossgmsdataadc0101b.irq.irq-PMI from timestamp 2018-08-06T17:25:02+00:00

Only change is passing endpoint argument to s3 script and if endpoint is specified connect using boto S3Connection. if not use the connect_to_region for bucket list and creation