Storage: local JSON file

[nazarewk]

This project seems like the only candidate so far to build a simple local IPAM (with "database" checked into git repository), but it was missing this type of storage, so I have implemented it as a local JSON file:

• uses internal Memory backend for operating on the data
  • preserved it's internal data structure in the output
• "refreshing" from file on reads and fully reloading on writes based on tracking of modification times
• json.go for serialization and deserialization of Prefixes

It is pretty barebones and could be improved, but should be good enough for easy evaluation of the project/library with persistence enabled.

some improvements I thought of:

• fix Dump()/Load() losing some informations (eg: Namespaces) and reuse this logic as a JSON data structure
• could involve file system locking of files (didn't go with it at the start because it's pretty unix'y feature)
• could be more performant:
  • split into multiple files for smaller edits
  • more efficient reload (eg: diff the data instead of clearing and loading whole file)

[majst01]

Nice addition, had only a first look for cosmetic stuff, deeper look later

[majst01]

Linter complains:

  Error: G306: Expect WriteFile permissions to be 0600 or less (gosec)
  Error: type `testableFileStorage` is unused (unused)
  Error: type assertion on error will fail on wrapped errors. Use errors.As to check for specific errors (errorlint)
  Error: comparing with != will fail on wrapped errors. Use errors.Is to check for a specific error (errorlint)
  
  Error: issues found

[majst01]

I think there is nothing wrong with your approach, but have you considered using SQLite instead, this would also be a local storage solution but with all the downsides you mentioned fixed ?

I started such a support back then, but havent enough time to finish that.

[nazarewk]

Yeah my first thought was SQLite, but it would not advance in any meaningful way my use case of checking state into the repository in a human readable format.

[nazarewk]

not sure what is this error ERROR: denied: installation not allowed to Write organization package

[majst01]

not sure what is this error ERROR: denied: installation not allowed to Write organization package

PRs from outside the metal-stack org are not allowed to push to our registry, this is nothing to worry about. We where not able to implement this in the CI that the push is not executed in such a case.

[codecov-commenter]

Codecov Report

Merging #123 (04f4405) into master (6f010ba) will decrease coverage by 0.73%.
The diff coverage is 57.57%.

❗ Current head 04f4405 differs from pull request most recent head bb6d2bd. Consider uploading reports for the commit bb6d2bd to get more accurate results

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master     #123      +/-   ##
==========================================
- Coverage   64.85%   64.13%   -0.73%     
==========================================
  Files          10       11       +1     
  Lines        1491     1656     +165     
==========================================
+ Hits          967     1062      +95     
- Misses        341      380      +39     
- Partials      183      214      +31     

Files Changed	Coverage Δ
file.go	57.57% <57.57%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[majst01]

Nice, its still quite fast:

BenchmarkNewPrefix/MongoDB-2    	     662	   1790806 ns/op	   34140 B/op	     397 allocs/op
BenchmarkAcquireIP/Memory-2     	  255164	      4491 ns/op	    2224 B/op	      43 allocs/op
BenchmarkAcquireIP/File-2       	    2041	    543635 ns/op	   20134 B/op	     210 allocs/op
BenchmarkAcquireIP/Postgres-2   	     310	   3839610 ns/op	   10969 B/op	     262 allocs/op
BenchmarkAcquireIP/Cockroach-2  	     135	   9741293 ns/op	   10964 B/op	     261 allocs/op
BenchmarkAcquireIP/Redis-2      	     921	   1340158 ns/op	   11822 B/op	     272 allocs/op
BenchmarkAcquireIP/Etcd-2       	     345	   3552269 ns/op	   63949 B/op	    1073 allocs/op
BenchmarkAcquireIP/KeyDB-2      	     872	   1487534 ns/op	   11820 B/op	     272 allocs/op
BenchmarkAcquireIP/MongoDB-2    	     651	   1842363 ns/op	   40701 B/op	     532 allocs/op

Can you add the filesytem backend into the Readme, skip the benchmark numbers for now.

[nazarewk]

Nice, its still quite fast:

BenchmarkNewPrefix/MongoDB-2    	     662	   1790806 ns/op	   34140 B/op	     397 allocs/op
BenchmarkAcquireIP/Memory-2     	  255164	      4491 ns/op	    2224 B/op	      43 allocs/op
BenchmarkAcquireIP/File-2       	    2041	    543635 ns/op	   20134 B/op	     210 allocs/op
BenchmarkAcquireIP/Postgres-2   	     310	   3839610 ns/op	   10969 B/op	     262 allocs/op
BenchmarkAcquireIP/Cockroach-2  	     135	   9741293 ns/op	   10964 B/op	     261 allocs/op
BenchmarkAcquireIP/Redis-2      	     921	   1340158 ns/op	   11822 B/op	     272 allocs/op
BenchmarkAcquireIP/Etcd-2       	     345	   3552269 ns/op	   63949 B/op	    1073 allocs/op
BenchmarkAcquireIP/KeyDB-2      	     872	   1487534 ns/op	   11820 B/op	     272 allocs/op
BenchmarkAcquireIP/MongoDB-2    	     651	   1842363 ns/op	   40701 B/op	     532 allocs/op

Can you add the filesytem backend into the Readme, skip the benchmark numbers for now.

Done, I guess it highly depends on the backing storage and it currently lands on a ramdisk (tmpfs for /tmp/)

[majst01]

Maybe give some contextual message to the error

[majst01]

Thanks for the contribution !