Skip to content

Commit

Permalink
Merge tag 'v1.1'
Browse files Browse the repository at this point in the history
- New maintainer: Kirill Müller.

- RSQLite always builds with the included source, which is located in `src/sqlite3`. This prevents bugs due to API mismatches and considerably simplifies the build process.

- Current version: 3.11.1.

- Enable JSON1 extension (#152, @TigerToes).

- Include support for FTS5 (@mkuhn).

- Compilation limits `SQLITE_MAX_VARIABLE_NUMBER` and `SQLITE_MAX_COLUMN` have been reset to the defaults. The documentation suggests setting to such high values is a bad idea.

- Header files for `sqlite3` are no longer installed, linking to the package is not possible anymore. Packages that require access to the low-level sqlite3 API should bundle their own copy.

- `RSQLite()` no longer automatically attaches DBI when loaded. This is to
  encourage you to use `library(DBI); dbConnect(RSQLite::SQLite())`.

- Functions that take a table name, such as `dbWriteTable()` and `dbReadTable()`,
  now quote the table name via `dbQuoteIdentifier()`.
  This means that caller-quoted names should be marked as such with `DBI::SQL()`.

- RSQLite has been rewritten (essentially from scratch) in C++ with
  Rcpp. This has considerably reduced the amount of code, and allows us to
  take advantage of the more sophisticated memory management tools available in
  Rcpp. This rewrite should yield some minor performance improvements, but
  most importantly protect against memory leaks and crashes. It also provides
  a better base for future development. In particular, it is now technically
  possible to have multiple result sets per connection, although this feature
  is currently disabled (#150).

- You can now use SQLite's URL specification for databases. This allows you to
  create [shared in-memory](https://www.sqlite.org/inmemorydb.html) databases
  (#70).

- Queries (#69), query parameters and table data are always converted to UTF-8 before being sent to the database.

- Adapted to `DBI` 0.5, new code should use `dbExecute()` instead of `dbGetQuery()`, and `dbSendStatement()` instead of `dbSendQuery()` where appropriate.

- New strategy for prepared queries. Create a prepared query with `dbSendQuery()` or `dbSendStatement()` and bind values with `dbBind()`. The same query/statement can be executed efficiently multiple times by passing a data-frame-like object (#168, #178, #181).

- `dbSendQuery()`, `dbGetQuery()`, `dbSendStatement()` and `dbExecute()`
  also support inline parameterised queries,
  like `dbGetQuery(datasetsDb(), "SELECT * FROM mtcars WHERE cyl = :cyl",
  params = list(cyl = 4))`. This has no performance benefits but protects you
  from SQL injection attacks.

- Improve column type inference: the first non-`NULL` value decides the type of a column (#111). If there are no non-`NULL` values, the column affinity is used, determined according to sqlite3 rules (#160).

- `dbFetch()` uses the same row name strategy as `dbReadTable()` (#53).

- `dbColumnInfo()` will now return information even before you've retrieved any data.

- New `sqliteVersion()` prints the header and library versions of RSQLite.

- Deprecation warnings are given only once, with a clear reference to the source.

- `datasetsDb()` now returns a read-only database, to avoid modifications to the installed file.

- `make.db.names()` has been formally deprecated. Please use `dbQuoteIdentifier()` instead. This function is also used in `dbReadTable()`, `dbRemoveTable()`, and `dbListFields()` (#106, #132).

- `sqliteBuildTableDefinition()` has been deprecated. Use `DBI::sqlCreateTable()` instead.

- `dbGetException()` now raises a deprecation warning and always returns `list(errorNum = 0L, errorMsg = "OK")`, because querying the last SQLite error only works if an error actually occurred (#129).

- `dbSendPreparedQuery()` and `dbGetPreparedQuery()` have been reimplemented (with deprecation warning) using `dbSendQuery()`, `dbBind()` and `dbFetch()` for compatibility with existing packages (#100, #153, #168, #181). Please convert to the new API, because the old function may be removed completely very soon: They were never part of the official API, and do less argument checking than the new APIs. Both `dbSendPreparedQuery()` and `dbGetPreparedQuery()` ignore parameters not found in the query, with a warning (#174).

- Reimplemented `dbListResults()` (with deprecation warning) for compatibility with existing packages (#154).

- Soft-deprecated `dbGetInfo()`: The "Result" method is implemented by DBI, the methods for the other classes raise a warning (#137). It's now better to access the metadata with individual functions `dbHasCompleted()`, `dbGetRowCount()` and `dbGetRowsAffected()`.

- All `summary()` methods have been removed: the same information is now displayed in the `show()` methods, which were previously pretty useless.

- The `raw` data type is supported in `dbWriteTable()`, creates a `TEXT` column with a warning (#173).

- Numeric values for the `row.names` argument are converted to logical, with a warning (#170).

- If the number of data frame columns matches the number of existing columns for `dbWriteTable(append = TRUE)`, columns will be matched by position for compatibility, with a warning in case of a name mismatch (#164).

- `dbWriteTable()` supports the `field.types` argument when creating a new table (#171), and the `temporary` argument, default `FALSE` (#113).

- Reexporting `dbGetQuery()` and `dbDriver()` (#147, #148, #183).

- `sqliteCopyDatabase()` accepts character as `to` argument again, in this case a temporary connection is opened.

- Reimplemented `dbWriteTable("SQLiteConnection", "character", "character")` for import of CSV files, using a function from the old codebase (#151).

- `dbWriteTable("SQLiteConnection", "character", "data.frame")` looks
  for table names already enclosed in backticks and uses these,
  (with a warning), for compatibility with the sqldf package.

- The `dbExistsTable()` function now works faster by filtering the list of tables using SQL (#166).

- Start on a basic vignette: `vignette("RSQLite")` (#50).

- Reworked function and method documentation, removed old documentation (#121).

- Using `dbExecute()` in documentation and examples.

- Using both `":memory:"` and `":file::memory:"` in documentation.

- Added additional documentation and unit tests for
  [autoincrement keys](https://www.sqlite.org/autoinc.html) (#119, @wibeasley).

- Avoid warning about missing `long long` data type in C++98 by using a compound data type built from two 32-bit integers, with static assert that the size is 8 indeed.

- Remove all compilation warnings.

- All DBI methods contain an ellipsis `...` in their signature. Only the `name` argument to the transaction methods appears before the ellipsis for compatibility reasons.

- Using the `DBItest` package for testing (#105), with the new `constructor_relax_args` tweak.

- Using the `plogr` for logging at the C++ level, can be enabled via `RSQLite:::init_logging()`.

- Using new `sqlRownamesToColumn()` and `sqlColumnToRownames()` (r-dbi/DBI#91).

- Using `astyle` for code formatting (#159), also in tests (but only if sources can be located), stripped space at end of line in all source files.

- Tracking dependencies between source and header files (#138).

- Moved all functions from headers to modules (#162).

- Fixed all warnings in tests (#157).

- Checking message wording for deprecation warnings (#157).

- Testing simple and named transactions (#163).

- Using container-based builds and development version of `testthat` on Travis.

- Enabled AppVeyor testing.

- Differential reverse dependency checks.

- Added upgrade script for sqlite3 sources and creation script for the datasets database to the `data-raw` directory.
  • Loading branch information
krlmlr committed Nov 30, 2016
2 parents 750624b + 6c173c6 commit e780af4
Show file tree
Hide file tree
Showing 23 changed files with 453 additions and 5,535 deletions.
3 changes: 2 additions & 1 deletion .Rbuildignore
Expand Up @@ -6,7 +6,6 @@ a.out.dSYM
^.*\.Rproj$
^\.Rproj\.user$
^\.travis\.yml$
^NEWS\.md$
^cran-comments\.md$
^data-raw$
^src/sqlite3/.*\.o$
Expand All @@ -26,3 +25,5 @@ a.out.dSYM
^clion-test\.R$
^_pkgdown\.yml$
^docs$
^README\.Rmd$
^UPDATE\.md$
2 changes: 1 addition & 1 deletion .travis.yml
Expand Up @@ -19,7 +19,7 @@ matrix:
- r: oldrel
- r: devel
- os: osx
osx_image: xcode7.2
osx_image: xcode7.3
r: release
latex: false

Expand Down
23 changes: 12 additions & 11 deletions DESCRIPTION
@@ -1,38 +1,38 @@
Package: RSQLite
Version: 1.0.9017
Date: 2016-11-22
Title: SQLite Interface for R
Version: 1.1
Date: 2016-11-24
Title: 'SQLite' Interface for R
Authors@R: c(
person("Kirill", "Müller", role = c("aut", "cre"), email = "krlmlr+r@mailbox.org"),
person("Hadley", "Wickham", role = c("aut")),
person(c("David", "A."), "James", role = "aut"),
person("Seth", "Falcon", role = "aut"),
person(family = "SQLite Authors", role = "ctb", comment = "for the included SQLite sources"),
person("Liam", "Healy", role = "ctb", comment = "for the included SQLite sources"),
person(family = "R Consortium", role = "cph"),
person(family = "RStudio", role = "cph")
)
Description: Embeds the 'SQLite' database engine in R and
provides an interface compliant with the 'DBI' package. The
source for the SQLite engine (version 3.8.8.2) is included.
source for the 'SQLite' engine (version 3.8.8.2) is included.
Depends:
R (>= 3.1.0)
Suggests:
testthat,
DBItest,
knitr,
rmarkdown,
DBItest
testthat
Imports:
methods,
DBI (>= 0.4-9),
Rcpp (>= 0.12.7),
memoise
memoise,
methods,
Rcpp (>= 0.12.7)
LinkingTo: Rcpp, BH, plogr
Remotes: rstats-db/DBI, rstats-db/DBItest@production, hadley/testthat
Encoding: UTF-8
License: LGPL (>= 2)
URL: https://github.com/rstats-db/RSQLite
BugReports: https://github.com/rstats-db/RSQLite/issues
Collate:
Collate:
'RcppExports.R'
'SQLiteConnection.R'
'SQLiteDriver.R'
Expand All @@ -41,6 +41,7 @@ Collate:
'copy.R'
'datasetsDb.R'
'deprecated.R'
'dummy.R'
'extensions.R'
'query.R'
'rownames.R'
Expand Down
1 change: 1 addition & 0 deletions NAMESPACE
Expand Up @@ -59,4 +59,5 @@ exportMethods(sqlData)
import(DBI)
import(methods)
importFrom(Rcpp,sourceCpp)
importFrom(memoise,memoise)
useDynLib(RSQLite)
26 changes: 19 additions & 7 deletions NEWS.md
@@ -1,3 +1,5 @@
# RSQLite 1.1 (2016-11-25)

- New maintainer: Kirill Müller.

## Bundled SQLite
Expand All @@ -14,6 +16,15 @@

- Header files for `sqlite3` are no longer installed, linking to the package is not possible anymore. Packages that require access to the low-level sqlite3 API should bundle their own copy.

## Breaking changes

- `RSQLite()` no longer automatically attaches DBI when loaded. This is to
encourage you to use `library(DBI); dbConnect(RSQLite::SQLite())`.

- Functions that take a table name, such as `dbWriteTable()` and `dbReadTable()`,
now quote the table name via `dbQuoteIdentifier()`.
This means that caller-quoted names should be marked as such with `DBI::SQL()`.

## New features

- RSQLite has been rewritten (essentially from scratch) in C++ with
Expand Down Expand Up @@ -51,10 +62,7 @@

- Deprecation warnings are given only once, with a clear reference to the source.

## Breaking changes

- `RSQLite()` no longer automatically attaches DBI when loaded. This is to
encourage you to use `library(DBI); dbConnect(RSQLite::SQLite())`.
- `datasetsDb()` now returns a read-only database, to avoid modifications to the installed file.

## Deprecated functions

Expand Down Expand Up @@ -88,6 +96,10 @@

- Reimplemented `dbWriteTable("SQLiteConnection", "character", "character")` for import of CSV files, using a function from the old codebase (#151).

- `dbWriteTable("SQLiteConnection", "character", "data.frame")` looks
for table names already enclosed in backticks and uses these,
(with a warning), for compatibility with the sqldf package.

## Performance

- The `dbExistsTable()` function now works faster by filtering the list of tables using SQL (#166).
Expand All @@ -96,15 +108,15 @@

- Start on a basic vignette: `vignette("RSQLite")` (#50).

- Using `dbExecute()` in examples.
- Reworked function and method documentation, removed old documentation (#121).

- Using `dbExecute()` in documentation and examples.

- Using both `":memory:"` and `":file::memory:"` in documentation.

- Added additional documentation and unit tests for
[autoincrement keys](https://www.sqlite.org/autoinc.html) (#119, @wibeasley).

- Removed old documentation (#121).

## Internal

- Avoid warning about missing `long long` data type in C++98 by using a compound data type built from two 32-bit integers, with static assert that the size is 8 indeed.
Expand Down
16 changes: 16 additions & 0 deletions R/dummy.R
@@ -0,0 +1,16 @@
#' Dummy methods
#'
#' Define here so that these methods can also be imported from this package.
#' Recommended practice is to import all methods from \pkg{DBI}.
#'
#' @name dummy-methods
#' @keywords internal
NULL

#' @rdname dummy-methods
#' @aliases dbGetQuery,NULL,ANY-method
#' @inheritParams DBI::dbGetQuery
#' @export
setMethod("dbGetQuery", "NULL", function(conn, statement, ...) {
stop("conn cannot be NULL", call. = FALSE)
})
1 change: 0 additions & 1 deletion R/query.R
Expand Up @@ -112,7 +112,6 @@ setMethod("dbSendQuery", c("SQLiteConnection", "character"),


#' @export
#' @rawNamespace exportMethods(dbGetQuery)
DBI::dbGetQuery


Expand Down
11 changes: 11 additions & 0 deletions R/table.R
Expand Up @@ -65,6 +65,8 @@ setMethod("dbWriteTable", c("SQLiteConnection", "character", "data.frame"),
if (overwrite && append)
stop("overwrite and append cannot both be TRUE", call. = FALSE)

name <- check_quoted_identifier(name)

row.names <- compatRowNames(row.names)

dbBegin(conn, "dbWriteTable")
Expand Down Expand Up @@ -270,6 +272,15 @@ string_to_utf8 <- function(value) {
value
}

check_quoted_identifier <- function(name) {
if (class(name)[[1L]] != "SQL" && grepl("^`.*`$", name)) {
warning_once("Quoted identifiers should have class SQL, use DBI::SQL() if the caller performs the quoting.")
name <- SQL(name)
}

name
}


#' Read a database table
#'
Expand Down
3 changes: 2 additions & 1 deletion R/utils.R
Expand Up @@ -14,4 +14,5 @@ warningc <- function(...) {
warning(..., call. = FALSE, domain = NA)
}

warning_once <- memoise::memoise(warningc)
#' @importFrom memoise memoise
warning_once <- memoise(warningc)
94 changes: 28 additions & 66 deletions cran-comments.md
@@ -1,79 +1,41 @@
The following notes were generated across my local OS X install and ubuntu running on travis-ci. Response to NOTEs across three platforms below.
## Test environments
* Ubuntu 16.10 (local install), R 3.3.2
* Ubuntu 12.04 (on travis-ci), R devel, release, and oldrel
* OS X (on travis-ci), R release
* win-builder (devel and release)

* I am taking over maintenance of RSQLite. I'ved asked Seth Falcon to
email you to confirm this.
## R CMD check results

* checking CRAN incoming feasibility ... NOTE
Possibly mis-spelled words in DESCRIPTION:
DBI (14:46)
SQLite (3:8, 12:46, 13:29, 15:24)

These are correct spellings
0 errors | 0 warnings | 2 notes

* checking compiled code ... NOTE
File ‘/Users/hadley/Documents/databases/RSQLite.Rcheck/RSQLite/libs/RSQLite.so’:
Found ‘___stderrp’, possibly from ‘stderr’ (C)
Object: ‘sqlite-all.o’
* New maintainer: Kirill Müller, previous maintainer: Hadley Wickham.
See https://github.com/rstats-db/RSQLite/commit/1cfbce07c678de#commitcomment-18978172
for a comment that indicates Hadley's consent.

This is in C code from the embedded SQLite database.

I also ran R CMD check on all downstream dependencies on R-devel. All packages that I install past R CMD check without ERRORs, WARNINGs, or NOTE. (See below for packages that I couldn't install). I informed all downstream maintainers of the pending update one month ago, giving plenty of time for fixes.
* Installed size: This package comes with a bundled RSQLite library.

Downstream dependency failure ---------------------------------------------------

CITAN ===================================================================
* checking package dependencies ... ERROR
Package required but not available: ‘RGtk2’
## Reverse dependencies

See the information on DESCRIPTION files in the chapter ‘Creating R
packages’ of the ‘Writing R Extensions’ manual.
Checked all 117 CRAN and BioConductor reverse dependencies on Ubuntu 16.04.
A few CRAN packages show errors in this version but succeed with RSQLite v1.0.0:

marmap ==================================================================
* checking package dependencies ... ERROR
Package required but not available: ‘ncdf’
- ecd (0.8.2) imports RSQLite <= 1.0.0, contacted maintainer several times,
last time on Oct 20

See the information on DESCRIPTION files in the chapter ‘Creating R
packages’ of the ‘Writing R Extensions’ manual.
- poplite (0.99.16), errors fixed in development version:
https://github.com/dbottomly/poplite/issues/2

MUCflights ==============================================================
* checking package dependencies ... ERROR
Package required but not available: ‘XML’
- tigreBrowserWriter (0.1.2), contacted maintainer on Nov 17, no response so far.
The code is using a prepared SQL query with more parameters than the query expects.
RSQLite is now intentially stricter about such errors, and I believe that a
downstream fix will be beneficial for the tigreBrowserWriter package.

See the information on DESCRIPTION files in the chapter ‘Creating R
packages’ of the ‘Writing R Extensions’ manual.

pitchRx =================================================================
* checking package dependencies ... ERROR
Package required but not available: ‘XML2R’

Package suggested but not available for checking: ‘rgl’

See the information on DESCRIPTION files in the chapter ‘Creating R
packages’ of the ‘Writing R Extensions’ manual.

rangeMapper =============================================================
* checking package dependencies ... ERROR
Package required but not available: ‘rgdal’

Package suggested but not available for checking: ‘rgeos’

See the information on DESCRIPTION files in the chapter ‘Creating R
packages’ of the ‘Writing R Extensions’ manual.

RQDA ====================================================================
* checking package dependencies ... ERROR
Packages required but not available: ‘gWidgetsRGtk2’ ‘RGtk2’

Package which this enhances but not available for checking: ‘rjpod’

See the information on DESCRIPTION files in the chapter ‘Creating R
packages’ of the ‘Writing R Extensions’ manual.

snplist =================================================================
* checking package dependencies ... ERROR
Package required but not available: ‘biomaRt’

See the information on DESCRIPTION files in the chapter ‘Creating R
packages’ of the ‘Writing R Extensions’ manual.
I couldn't check RQDA and vmsbase, they consistently fail checks with both
versions of RSQLite on my system but succeed checks on CRAN.

All other packages are consistent in their checking behavior for both RSQLite 1.0.0
and 1.1.

See https://github.com/rstats-db/RSQLite/blob/r-1.1/revdep/README.md for check results and https://github.com/rstats-db/RSQLite/blob/r-1.1/revdep/problems.md
for results for packages with problems only.
24 changes: 24 additions & 0 deletions man/dummy-methods.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit e780af4

Please sign in to comment.