/
ChangeLog
310 lines (235 loc) · 12.7 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
2022-06-21: Jamie Lentin <lentinj@shuttlethread.com>
Version 7.3-1
* Improved numerical accuracy when calculating weighted mean/stddev
2022-06-07: Jamie Lentin <lentinj@shuttlethread.com>
Version 7.3-0
* More helpful error messages if taxonomy (e.g. vessel) items missing
* Support DuckDB 0.3.4 (unit tests weren't order agnostic)
2021-11-21: Jamie Lentin <lentinj@shuttlethread.com>
Version 7.2-0
* Fixes to support DuckDB 0.3.1 (Bugs in dbWriteTable type casting)
* Add mfdb_dplyr_table(), for querying any table
* Fix selecting from taxonomy tables via. dplyr
2021-10-09: Jamie Lentin <lentinj@shuttlethread.com>
Version 7.1-1
* Support DuckDB 0.2.9 and 0.3.0 (fix tests to be NULLS first/last agnostic)
2021-08-18: Jamie Lentin <lentinj@shuttlethread.com>
Version 7.1-0
* mfdb_dplyr_* now supports nested taxonomy columns, e.g. trip_start_port_latitude
* DuckDB support. Specify mfdb('db_filename.duckdb')
* SQLite support. Specify mfdb('db_filename.sqlite')
* Use bulk_copy to speed up large taxonomy imports
Schema changes:
* Remove taxonomy name restrictions
* Change prey.count from INT to DOUBLE
2021-02-23: Jamie Lentin <lentinj@shuttlethread.com>
Version 7.0-0
* Add scale_index to mfdb_survey_index_mean & mfdb_survey_index_total
* Add 'area_size' as a possible scale_index value
* Switch to requiring RPostgres as the default DBI driver
* Allow DB params to be supplied with MFDB_* environment variables
* Don't require divisions to be made when querying by areacell
Schema changes:
* Age now stored as NUMERIC
* Add trip taxonomy
* Add port taxonomy, used by trip for start/end points
* Add population taxonomy, for stock groupings
* Add vessel_owner taxonomy
* Add liver/gonad/stomach weight columns as well as overall weight
* Add bait_type/net_type/hook_count/net_count/mesh_size to tow table
* Add depth to areacell table
* Change schema.count from REAL to DOUBLE
(unreleased) Jamie Lentin <lentinj@shuttlethread.com>
* Preserve precision when calculating total weight
* Inline small mfdb_groups to queries, improving performance
* Rewrite Rpath support to use Rpath::create.rpath.params
* Rollback transactions after user interrupts
* Expose open_ended in intervals for gadget3
* Renumber all taxonomy IDs if there's an overlap with existing ones (not just the overlapped range)
2020-04-10: Jamie Lentin <lentinj@shuttlethread.com>
Version 6.2-0
* Documentation example overhaul: Now ~all can be run given a Postgres database
* Add single_matches_only option to mfdb_find_species()
* Turn down logging level
2020-03-02 Jamie Lentin <lentinj@shuttlethread.com>
Version 6.1-1
* Minor bugfixes
2020-02-26 Jamie Lentin <lentinj@shuttlethread.com>
Version 6.1-0
* Produce same seeded bootstrap group output on 3.6.0+
* Postgres 11: Don't try checking if existing function is aggregate
* Compare REAL to REAL when dividing items up into intervals
2017-10-27 Jamie Lentin <lentinj@shuttlethread.com>
Version 6.0-0
* Add mfdb_na_group function to put NAs in the their own group
* Remove remaining cs_taxonomy differences, simplify code
* mfdb_sample_rawdata now will group into a list, like other sample functions
Schema changes:
* age now stored as REAL
* areacell size now stored as REAL
* Allow gear taxonomy to be changed
* Allow the species schema to be changed also
* Add tow_end_latitude/tow_end_longitude/tow_start/tow_duration
* Add gear_mesh_size/gear_mesh_size_min/gear_mesh_size_max
(unreleased) Jamie Lentin <lentinj@shuttlethread.com>
* Add parameters/suitiability support to surveydistribution
2017-03-14 Jamie Lentin <lentinj@shuttlethread.com>
Version 5.2-0
* Add like and not_like to mfdb_unaggregated #46
* Make upper/lower include tail on final group, don't add additional groups to end
* Allow open_ended at either end of an mfdb_interval / mfdb_step_interval by specifying c('upper', 'lower')
2015-10-25 Jamie Lentin <lentinj@shuttlethread.com>
Version 5.1-0
* destroy_schema = TRUE should drop the whole schema, not just the tables within
* Add mfdb_share_with, for sharing case studies on the MFDB schema
* Force tables to be created within schema
* Document use of MFDB server
* Force ssl/user/password when connecting to mfdb database
* Let mfdb_import_area import divisions, so mfdb_import_division becomes unnecessary
* Add dplyr interface
* Allow EwE output to be generated without vessel data
* When presented with overlapping IDs in vocabularies remap the ID.
* mfdb_area_size can be used to get the entire area size
* Add EwE matrix output functions
2016-05-20 Jamie Lentin <lentinj@shuttlethread.com>
Version 5.0-0
Schema changes:
* Break up case_study's into schemata, ditch case_study_id:
This is a fairly major behind the scenes change. Postgres separates tables
into schemas. Before MFDB put everything into the default "public" schema.
Now MFDB will use a different schema for each case study (i.e. the
parameter you give to mfdb()).
Whereas before there was a list of case studies that could be used, now
you can specify any name to use as a case study. Note that the names are
case-insensitive, so "North-Sea", "north_sea", "North-sea" are all
considered the same name.
This also means you can use pg_dump and pg_restore to backup and restore
(or transfer to a different computer) individual case studies. For
examples, see:
http://mareframe.github.io/mfdb/reference/mfdb-package.html
Finally, the mfdb(destroy_schema = TRUE) option now expects a case study
/ schema name, since it is able to remove just one schema instead of
removing everything.
To make upgrades seamless, MFDB 5.x will, if the schema is new, check the
"public" schema first to see if there is any existing MFDB 4.x data stored
under the same case study name. If so it will copy the data into a new
schema. In theory once it's done this for any case studies you use you
can drop the tables in the public schema to save space, although make
sure your data is copied correctly first.
If you look at MFDB data via. psql, then you will need to make sure you
either set the default schema with "SET search_path TO north_sea", or
specify the schema in queries, e.g. "SELECT * FROM north_sea.sample".
2016-06-10 Jamie Lentin <lentinj@shuttlethread.com>
Version 4.1-0
* Add mfdb_sample_scaled with ability to scale by tow_length
* mfdb_stomach_preymeanweight shoult take the mean of unique stomachs, ignoring prey restrictions
* Add mfdb_sample_rawdata to help debugging
* Add mfdb_stomach_preyweightratio to get ratio of prey by weight
* mfdb_stomach_preymeanweight should take the mean of unique stomachs
* mfdb_stomach_preymeanlength should ignore prey samples without count
* When importing stomachs, make the default prey count unknown
* When importing stomachs, use same weight_total option for prey as regular samples
* Add mfdb_concatenate_results to help when you want to concatenate 2 result sets
* Allow prey names & coefficients to be specified per-prey
* Predator length agg file should be based on the "predator_length" column
* Add Saduria Entomon to species list
* Add case study IDs for Atlantis models
* State which columns are available when cannot find column
2015-11-06 Jamie Lentin <lentinj@shuttlethread.com>
Version 4.0-0
* Scale results by tow_length as well as any survey indices
* Allow querying by t_group (i.e. country) as well as institute
Schema changes:
* Remove fleet taxonomy, can group vessels instead
* Split vessel taxonomy up into vessel and vessel_type
* Add tow taxonomy, with extra columns for tow details
* Add t_group to all taxonomies, for one level of nesting
* Prey numbers doesn't accept NA #18
2015-10-29 Jamie Lentin <lentinj@shuttlethread.com>
Version 3.3-0
* Implement surveydistribution likelihood component
* Better gadget_* error messages when column names don't match (or there's no data at all)
* Example for fecthing DATRAS Stomach data
* Add mfdb_find_species helper to map from other species lists
2015-09-21 Jamie Lentin <lentinj@shuttlethread.com>
Version 3.2-0
* Only the length aggfiles should be of the form (min, max)
* Add mfdb_survey_index_total
* Import functions should allow removal of data
* Add institutes for Baltic
* Add Deep-water rose shrimp to species list
* Add CNR-IAMC to institute list
* Remove need for specifying fleetname in data for gadget_fleet_component, it makes no sense
* mfdb_step_interval must be integer, it's not going to work otherwise
* Add support for fleet files
* Fix bootstrapping when multiple group columns are used
* Notice if a query function is given a non-list params
* Weighted stddev should ignore NULL rows
* Weighted mean should ignore NULL rows
2015-04-15 Jamie Lentin <lentinj@shuttlethread.com>
Version 3.1-0
* Fix weighted mean function when scaling by an index
* Make aggregationlevel in catchbykilos automatic
* Allow import of "weight_total", for total biomass
2015-03-16 Jamie Lentin <lentinj@shuttlethread.com>
Version 3.0-0
Schema changes:
* Allow non-integer counts to be stored, when they have been normalised by haul-length, e.g.
* Allow count to be NA, when storing biomass from landings
* Make fleet a CS taxonomy, so you can have custom fleets
2015-04-15 Jamie Lentin <lentinj@shuttlethread.com>
Version 2.1-0
* Implement weighted stddev aggregate function (ignoring _var columns for now)
* Use survey indices to scale count, not replace it
* Support grouping of taxonomy items
* Allow import/export to tarball as well as directory
* Don't write files in one go when bulk exporting
* Fix surveyindices likelihood component, should be able to output multiple stocknames
* Add gadget_stockfile_initialconditions and gadget_stockfile_recruitment
* Add gadget_stockfile_extremes and gadget_stockfile_refweight for manipulating stockfiles
* Allow files in subdirectories to be read and then rewritten in the same place
* Introduce "bare components" file type, use this to include temperature
2015-03-10 Jamie Lentin <lentinj@shuttlethread.com>
Version 2.0-0
* Bulk-load prey data, speeding up stomach import
* Should use name to compare new/old items in taxonomies, not id
* Give meaningful errors (or just sort it) when lists are handed to gadget_likelihood_function
* ANALYZE only our tables, so we don't trigger warnings
* Don't match partial names (e.g. age --> maturity_stage) when importing data
* Autodiscover min and max for 'all' columns
* Only recreate new temporary tables for groups once
* Use arbitary 1:100 for 'all' aggregation
* Correct spelling of surveyindices
* Add mfdb_survey_index_mean, for retrieving acoustic indicies
* Query functions should always return a list of data.frames, even if there is no data
* Implement catchinkilos likelihood component
* Give examples of invalid data if import fails
* Add surveyindicies, migrationpenalty likelihood components
* Allow specifying an abundance_index when fetching mean length
* Add survey_index table and index_type taxonomy, to store acoustic data
* Add mfdb_import_cs_taxonomy, to allow changing any of the taxonomies
* Remove the distinction between survey data and survey metadata
2015-01-14 Jamie Lentin <lentinj@shuttlethread.com>
Version 1.1-1
* Add path to Debian/Ubuntu Postgres socket
2015-01-09 Jamie Lentin <lentinj@shuttlethread.com>
Version 1.1-0
* Base aggreations for open intervals and unaggreated data on returned data
* Fix iceland example so we only include each subdivision once
* Add an example for using bootstrapping
* Inspect data.frame columns when making a likelihood component
* Allow writing to alternative likelihood files
* When grouping by vectors with taxonomies, don't return integer values
* Need temperature to 1 dp
* More notes on how to install under windows
* Rework mfdb()'s ability to guess where DB is
* Don't overload as.list for mfdb_aggregates, use an internal generic
function instead
* Fix sampling for bootstrap groups
* Should run pre_query and from_clause for filter_cols too, just in case
they are an mfdb_group
* Use DB to find out case study ID---might be a remote DB
* R CMD check fixes
2014-08-23 Jamie Lentin <lentinj@shuttlethread.com>
Version 1.0-0
Initial release.